2026-10-29 00:52:47,674 [ 159810 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:42, check_args_and_update_paths) 2026-10-29 00:52:47,674 [ 159810 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:86, check_args_and_update_paths) 2026-10-29 00:52:47,674 [ 159810 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:97, check_args_and_update_paths) 2026-10-29 00:52:47,674 [ 159810 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:99, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_tq4w3n --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=2993bc2bf171 -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS=" -rfEps --run-id=2 --color=no --durations=0 test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: xdist-3.5.0, random-0.2, reportlog-0.4.0, order-1.0.1, timeout-2.2.0, repeat-0.9.3 timeout: 900.0s timeout method: signal timeout func_only: False collecting ... collected 1 item test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries FAILED [100%] =================================== FAILURES =================================== ___________________ test_query_is_canceled_with_inf_retries ____________________ cluster = broken_s3 = def test_query_is_canceled_with_inf_retries(cluster, broken_s3): node = cluster.instances["node_with_inf_s3_retries"] broken_s3.setup_at_part_upload( count=10000000, after=2, action="connection_refused", ) insert_query_id = f"TEST_QUERY_IS_CANCELED_WITH_INF_RETRIES" request = node.get_query_request( f""" INSERT INTO TABLE FUNCTION s3( 'http://resolver:8083/root/data/test_query_is_canceled_with_inf_retries', 'minio', 'minio123', 'CSV', auto, 'none' ) SELECT * FROM system.numbers LIMIT 1000000 SETTINGS s3_max_single_part_upload_size=100, s3_min_upload_part_size=10000, s3_check_objects_after_upload=0 """, query_id=insert_query_id, ) assert_eq_with_retry( node, f"SELECT count() FROM system.processes WHERE query_id='{insert_query_id}'", "1", ) assert_eq_with_retry( node, f"SELECT ProfileEvents['S3WriteRequestsErrors'] > 10 FROM system.processes WHERE query_id='{insert_query_id}'", "1", retry_count=12, sleep_time=10, ) > node.query(f"KILL QUERY WHERE query_id = '{insert_query_id}' ASYNC") test_checking_s3_blobs_paranoid/test.py:544: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3456: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 243, stderr: Received exception from server (version 24.3.18): E Code: 499. DB::Exception: Received from 172.16.1.7:9000. DB::Exception: Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 24.3.18.10427.altinitystable (altinity build)). Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d2e1b23 E 1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000eff24f4 E 2. ./contrib/llvm-project/libcxx/include/string:1499: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x0000000015e52f5d E 3. ./build_docker/./src/Common/ThreadPoolTaskTracker.cpp:118: void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000015a06669 E 4. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x0000000015e0f700 E 5. ./contrib/llvm-project/libcxx/include/future:2069: std::packaged_task::operator()() @ 0x0000000011ee31d5 E 6. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x0000000015e0fbed E 7. ./src/Common/Stopwatch.h:26: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000f0d6392 E 8. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f0db6fd E 9. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0db662 E 10. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl::worker(std::__list_iterator) @ 0x000000000f0d1e20 E 11. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000f0d8612 E 12. __tsan_thread_start_func @ 0x000000000706b10f E 13. ? @ 0x00007fd90da41ac3 E 14. ? @ 0x00007fd90dad3850 E . (S3_ERROR) E (query: KILL QUERY WHERE query_id = 'TEST_QUERY_IS_CANCELED_WITH_INF_RETRIES' ASYNC) helpers/client.py:239: QueryRuntimeException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2026-10-28 23:52:50 [ 499 ] DEBUG : Command:['docker ps | wc -l'] (cluster.py:113, run_and_check) 2026-10-28 23:52:50 [ 499 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) 2026-10-28 23:52:50 [ 499 ] DEBUG : No running containers (conftest.py:92, cleanup_environment) 2026-10-28 23:52:50 [ 499 ] DEBUG : Pruning Docker networks (conftest.py:94, cleanup_environment) 2026-10-28 23:52:50 [ 499 ] DEBUG : Command:['docker network prune --force'] (cluster.py:113, run_and_check) 2026-10-28 23:52:50 [ 499 ] DEBUG : Command:["sysctl net.ipv4.ip_local_port_range='55000 65535'"] (cluster.py:113, run_and_check) 2026-10-28 23:52:50 [ 499 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:121, run_and_check) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV HOSTNAME 2953fa402359 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV SHLVL 0 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV HOME /root (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV OLDPWD / (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_HELPER_TAG 2cffe1eae894 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV PYTHONUNBUFFERED 1 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_PYTHON_BOTTLE_TAG a2d3dc777d0c (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV UBSAN_OPTIONS print_stacktrace=1 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV PYTEST_ADDOPTS -rfEps --run-id=2 --color=no --durations=0 test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries -vvv (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV COMPOSE_HTTP_TIMEOUT 600 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV PATH /spark-3.3.2-bin-hadoop3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_KERBERIZED_HADOOP_TAG ce74919e88f5 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_CHANNEL stable (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_CLIENT_TIMEOUT 300 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV PWD /ClickHouse/tests/integration (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV TZ Etc/UTC (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV DOCKER_BASE_TAG 2993bc2bf171 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV LC_CTYPE C.UTF-8 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV INTEGRATION_TESTS_RUN_ID 2 (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : ENV PYTEST_CURRENT_TEST test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries (setup) (cluster.py:400, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:696, __init__) 2026-10-28 23:52:50 [ 499 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1670, add_instance) 2026-10-28 23:52:50 [ 499 ] DEBUG : Cluster name: project_name:roottestcheckings3blobsparanoid. Added instance name:node tag:2993bc2bf171 base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env', '--project-name', 'roottestcheckings3blobsparanoid', '--file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:1952, add_instance) 2026-10-28 23:52:50 [ 499 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1670, add_instance) 2026-10-28 23:52:50 [ 499 ] DEBUG : Cluster name: project_name:roottestcheckings3blobsparanoid. Added instance name:node_with_inf_s3_retries tag:2993bc2bf171 base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env', '--project-name', 'roottestcheckings3blobsparanoid', '--file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node_with_inf_s3_retries/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:1952, add_instance) 2026-10-28 23:52:50 [ 499 ] INFO : Starting cluster... (test.py:37, cluster) 2026-10-28 23:52:50 [ 499 ] INFO : Running tests in /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/test.py (cluster.py:2659, start) 2026-10-28 23:52:50 [ 499 ] DEBUG : Cluster start called. is_up=False (cluster.py:2666, start) 2026-10-28 23:52:50 [ 499 ] DEBUG : Docker networks for project roottestcheckings3blobsparanoid are NETWORK ID NAME DRIVER SCOPE (cluster.py:780, print_all_docker_pieces) 2026-10-28 23:52:50 [ 499 ] DEBUG : Docker containers for project roottestcheckings3blobsparanoid are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:788, print_all_docker_pieces) 2026-10-28 23:52:50 [ 499 ] DEBUG : Docker volumes for project roottestcheckings3blobsparanoid are DRIVER VOLUME NAME (cluster.py:796, print_all_docker_pieces) 2026-10-28 23:52:50 [ 499 ] DEBUG : Cleanup called (cluster.py:801, cleanup) 2026-10-28 23:52:50 [ 499 ] DEBUG : Docker networks for project roottestcheckings3blobsparanoid are NETWORK ID NAME DRIVER SCOPE (cluster.py:780, print_all_docker_pieces) 2026-10-28 23:52:50 [ 499 ] DEBUG : Docker containers for project roottestcheckings3blobsparanoid are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:788, print_all_docker_pieces) 2026-10-28 23:52:50 [ 499 ] DEBUG : Docker volumes for project roottestcheckings3blobsparanoid are DRIVER VOLUME NAME (cluster.py:796, print_all_docker_pieces) 2026-10-28 23:52:50 [ 499 ] DEBUG : Command:docker container list --all --filter name='^/roottestcheckings3blobsparanoid_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-10-28 23:52:50 [ 499 ] DEBUG : Unstopped containers: {} (cluster.py:815, cleanup) 2026-10-28 23:52:50 [ 499 ] DEBUG : No running containers for project: roottestcheckings3blobsparanoid (cluster.py:829, cleanup) 2026-10-28 23:52:50 [ 499 ] DEBUG : Trying to prune unused networks... (cluster.py:835, cleanup) 2026-10-28 23:52:50 [ 499 ] DEBUG : Trying to prune unused images... (cluster.py:851, cleanup) 2026-10-28 23:52:50 [ 499 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2026-10-28 23:52:50 [ 499 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check) 2026-10-28 23:52:50 [ 499 ] DEBUG : Images pruned (cluster.py:854, cleanup) 2026-10-28 23:52:50 [ 499 ] DEBUG : Trying to prune unused volumes... (cluster.py:860, cleanup) 2026-10-28 23:52:50 [ 499 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-10-28 23:52:50 [ 499 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) 2026-10-28 23:52:50 [ 499 ] DEBUG : Setup directory for instance: node (cluster.py:2679, start) 2026-10-28 23:52:50 [ 499 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4383, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Create directory for common tests configuration (cluster.py:4388, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Copy common configuration from helpers (cluster.py:4408, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Generate and write macros file (cluster.py:4441, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/configs/storage_conf.xml'] to /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/configs/config.d (cluster.py:4471, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/database (cluster.py:4488, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/logs (cluster.py:4499, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4582, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Setup directory for instance: node_with_inf_s3_retries (cluster.py:2679, start) 2026-10-28 23:52:50 [ 499 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4383, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Create directory for common tests configuration (cluster.py:4388, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Copy common configuration from helpers (cluster.py:4408, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Generate and write macros file (cluster.py:4441, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/configs/storage_conf.xml'] to /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node_with_inf_s3_retries/configs/config.d (cluster.py:4471, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node_with_inf_s3_retries/database (cluster.py:4488, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node_with_inf_s3_retries/logs (cluster.py:4499, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4582, create_dir) 2026-10-28 23:52:50 [ 499 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/minio/certs/public.crt'} stored in /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env (cluster.py:86, _create_env_file) 2026-10-28 23:52:50 [ 499 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-28 23:52:50 [ 499 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-28 23:52:50 [ 499 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-28 23:52:50 [ 499 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-28 23:52:50 [ 499 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:546, _make_request) 2026-10-28 23:52:50 [ 499 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env', '--project-name', 'roottestcheckings3blobsparanoid', '--file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node_with_inf_s3_retries/docker-compose.yml', 'pull'] (cluster.py:113, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling proxy2 ... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling proxy1 ... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling minio1 ... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling node_with_inf_s3_retries ... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling node ... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling resolver ... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling proxy2 ... pulling from altinityinfra/s3-proxy (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling minio1 ... pulling from clickhouse/minio-minio (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling node_with_inf_s3_retries ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling resolver ... pulling from altinityinfra/python... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling proxy1 ... pulling from altinityinfra/s3-proxy (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling proxy2 ... digest: sha256:92e86c3de6b8ff00d8... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling proxy2 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling minio1 ... digest: sha256:9eea83a4e1425067e7... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling minio1 ... status: image is up to date for c... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling proxy2 ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling node_with_inf_s3_retries ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling node_with_inf_s3_retries ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling minio1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling node_with_inf_s3_retries ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling resolver ... digest: sha256:6004f5bd254dd16f55... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling resolver ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling resolver ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling proxy1 ... digest: sha256:92e86c3de6b8ff00d8... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling proxy1 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling proxy1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling node ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling node ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling node ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] DEBUG : Stderr:Pulling node ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:51 [ 499 ] INFO : Trying to create Minio instance by command docker-compose --env-file /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env --project-name roottestcheckings3blobsparanoid --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d (cluster.py:2959, start) 2026-10-28 23:52:51 [ 499 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env', '--project-name', 'roottestcheckings3blobsparanoid', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--verbose', 'up', '-d'] (cluster.py:113, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.config.config.find: Using configuration files: /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.docker_client.get_client: docker-compose version 1.29.2, build unknown (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:docker-py version: (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:CPython version: 3.10.12 (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:OpenSSL version: OpenSSL 3.0.2 15 Mar 2022 (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '23.0.6', 'Details': {'ApiVersion': '1.42', 'Arch': 'amd64', 'BuildTime': '2023-05-05T21:18:13.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9dbdbd4', 'GoVersion': 'go1.19.9', 'KernelVersion': '5.15.0-198-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.7.18', 'Details': {'GitCommit': 'ae71819c4f5e67bb4d5ae76a6b735f29cc25774e'}}, {'Name': 'runc', 'Version': '1.7.18', 'Details': {'GitCommit': 'v1.1.13-0-g58aa920'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=23.0.6, ApiVersion=1.42, MinAPIVersion=1.12, GitCommit=9dbdbd4, GoVersion=go1.19.9, Os=linux, Arch=amd64, KernelVersion=5.15.0-198-generic, BuildTime=2023-05-05T21:18:13.000000000+00:00 (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestcheckings3blobsparanoid_default') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('roottestcheckings3blobsparanoid_data1-1') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info <- () (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'BridgeNfIp6tables': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'BridgeNfIptables': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'CPUSet': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'CPUShares': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'CgroupDriver': 'cgroupfs', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'CgroupVersion': '2', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'ContainerdCommit': {'Expected': 'ae71819c4f5e67bb4d5ae76a6b735f29cc25774e', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'ID': 'ae71819c4f5e67bb4d5ae76a6b735f29cc25774e'}, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Containers': 0, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestcheckings3blobsparanoid_default') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.network.ensure: Creating network "roottestcheckings3blobsparanoid_default" with the default driver (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network <- (name='roottestcheckings3blobsparanoid_default', driver=None, options=None, ipam=None, internal=False, enable_ipv6=False, labels={'com.docker.compose.project': 'roottestcheckings3blobsparanoid', 'com.docker.compose.network': 'default', 'com.docker.compose.version': '1.29.2'}, attachable=True, check_duplicate=True) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network -> {'Id': 'dd8a82d83a44efc5f8c5e54df5cf3034f80d0aa677d95f125baccf2d69f4c18a', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Warning': ''} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('roottestcheckings3blobsparanoid_data1-1') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.volume.initialize: Creating volume "roottestcheckings3blobsparanoid_data1-1" with default driver (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_volume <- ('roottestcheckings3blobsparanoid_data1-1', None, None, labels={'com.docker.compose.project': 'roottestcheckings3blobsparanoid', 'com.docker.compose.volume': 'data1-1', 'com.docker.compose.version': '1.29.2'}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_volume -> {'CreatedAt': '2026-10-28T23:52:52Z', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Driver': 'local', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Labels': {'com.docker.compose.project': 'roottestcheckings3blobsparanoid', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'com.docker.compose.version': '1.29.2', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'com.docker.compose.volume': 'data1-1'}, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Mountpoint': '/var/lib/docker/volumes/roottestcheckings3blobsparanoid_data1-1/_data', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Name': 'roottestcheckings3blobsparanoid_data1-1', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Options': None, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Scope': 'local'} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['minio'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle:a2d3dc777d0c') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.project._get_convergence_plans: minio1 has upstream changes (proxy1, proxy2) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.project._get_convergence_plans: resolver has upstream changes (proxy1, proxy2) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, , , } (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_proxy1_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestcheckings3blobsparanoid', service='proxy1', number=1)} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestcheckings3blobsparanoid', service='proxy1', number=1) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_proxy2_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestcheckings3blobsparanoid', service='proxy2', number=1)} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestcheckings3blobsparanoid', service='proxy2', number=1) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 713ed5c3d8720f843a74b60c2bd587394134466a645c725e1bb6dbea435304b9 (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestcheckings3blobsparanoid_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'NetworkMode': 'roottestcheckings3blobsparanoid_default', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/s3-proxy', volumes={}, name='roottestcheckings3blobsparanoid_proxy1_1', detach=True, ports=['8080', '80', '443'], environment=[], labels={'com.docker.compose.project': 'roottestcheckings3blobsparanoid', 'com.docker.compose.service': 'proxy1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '713ed5c3d8720f843a74b60c2bd587394134466a645c725e1bb6dbea435304b9'}, host_config={'NetworkMode': 'roottestcheckings3blobsparanoid_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'roottestcheckings3blobsparanoid_default': {'Aliases': ['proxy1'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 713ed5c3d8720f843a74b60c2bd587394134466a645c725e1bb6dbea435304b9 (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestcheckings3blobsparanoid_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'NetworkMode': 'roottestcheckings3blobsparanoid_default', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/s3-proxy', volumes={}, name='roottestcheckings3blobsparanoid_proxy2_1', detach=True, ports=['8080', '80', '443'], environment=[], labels={'com.docker.compose.project': 'roottestcheckings3blobsparanoid', 'com.docker.compose.service': 'proxy2', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '713ed5c3d8720f843a74b60c2bd587394134466a645c725e1bb6dbea435304b9'}, host_config={'NetworkMode': 'roottestcheckings3blobsparanoid_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'roottestcheckings3blobsparanoid_default': {'Aliases': ['proxy2'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '74c017543899e7cdb0955a7d21c547647a195bde12f66131d42d3f1f4380eca2', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('74c017543899e7cdb0955a7d21c547647a195bde12f66131d42d3f1f4380eca2') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': 'f9b923c7ba8ff0b110d30ce54b6f1aa93beae9aec0edb19d24f33974f9b95d6a', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('f9b923c7ba8ff0b110d30ce54b6f1aa93beae9aec0edb19d24f33974f9b95d6a') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Args': ['/run.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'NGINX_VERSION=1.27.5', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('74c017543899e7cdb0955a7d21c547647a195bde12f66131d42d3f1f4380eca2', 'roottestcheckings3blobsparanoid_default') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Args': ['/run.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'NGINX_VERSION=1.27.5', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('f9b923c7ba8ff0b110d30ce54b6f1aa93beae9aec0edb19d24f33974f9b95d6a', 'roottestcheckings3blobsparanoid_default') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('f9b923c7ba8ff0b110d30ce54b6f1aa93beae9aec0edb19d24f33974f9b95d6a', 'roottestcheckings3blobsparanoid_default', aliases=['f9b923c7ba8f', 'proxy2'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('74c017543899e7cdb0955a7d21c547647a195bde12f66131d42d3f1f4380eca2', 'roottestcheckings3blobsparanoid_default', aliases=['74c017543899', 'proxy1'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('74c017543899e7cdb0955a7d21c547647a195bde12f66131d42d3f1f4380eca2') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('f9b923c7ba8ff0b110d30ce54b6f1aa93beae9aec0edb19d24f33974f9b95d6a') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestcheckings3blobsparanoid', service='proxy1', number=1) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_proxy1_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestcheckings3blobsparanoid', service='proxy2', number=1) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_proxy2_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestcheckings3blobsparanoid', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_resolver_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestcheckings3blobsparanoid', service='resolver', number=1)} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestcheckings3blobsparanoid', service='resolver', number=1) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_minio1_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestcheckings3blobsparanoid', service='minio1', number=1)} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestcheckings3blobsparanoid', service='minio1', number=1) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle:a2d3dc777d0c') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['minio'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle:a2d3dc777d0c') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['minio'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 493adae22b486ec424d14c2bf233d9f842bbd6a44dba9c2c32430dec7e8dc1a4 (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=['roottestcheckings3blobsparanoid_data1-1:/data1:rw', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/minio/certs:/certs:rw'], volumes_from=[], privileged=False, network_mode='roottestcheckings3blobsparanoid_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': ['roottestcheckings3blobsparanoid_data1-1:/data1:rw', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/minio/certs:/certs:rw'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'NetworkMode': 'roottestcheckings3blobsparanoid_default', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (command='server --console-address 127.0.0.1:19001 --address :9001 --certs-dir /certs /data1-1', environment=['MINIO_ACCESS_KEY=minio', 'MINIO_SECRET_KEY=minio123', 'MINIO_PROMETHEUS_AUTH_TYPE=public'], image='clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z', volumes={'/data1': {}, '/certs': {}}, name='roottestcheckings3blobsparanoid_minio1_1', detach=True, ports=['9001'], labels={'com.docker.compose.project': 'roottestcheckings3blobsparanoid', 'com.docker.compose.service': 'minio1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '493adae22b486ec424d14c2bf233d9f842bbd6a44dba9c2c32430dec7e8dc1a4'}, host_config={'NetworkMode': 'roottestcheckings3blobsparanoid_default', 'VolumesFrom': [], 'Binds': ['roottestcheckings3blobsparanoid_data1-1:/data1:rw', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/minio/certs:/certs:rw'], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'roottestcheckings3blobsparanoid_default': {'Aliases': ['minio1'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: d620dabbb355723a9d632ad45f7b3b6fabe85fbe0149372146546b6429792649 (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestcheckings3blobsparanoid_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'NetworkMode': 'roottestcheckings3blobsparanoid_default', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/python-bottle:a2d3dc777d0c', tty=True, volumes={}, name='roottestcheckings3blobsparanoid_resolver_1', detach=True, ports=['8080'], environment=[], labels={'com.docker.compose.project': 'roottestcheckings3blobsparanoid', 'com.docker.compose.service': 'resolver', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': 'd620dabbb355723a9d632ad45f7b3b6fabe85fbe0149372146546b6429792649'}, host_config={'NetworkMode': 'roottestcheckings3blobsparanoid_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'roottestcheckings3blobsparanoid_default': {'Aliases': ['resolver'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '6fb31558cdddef0d1f14cdc872dc179d928214555dd589cc6f235ac06f4706d7', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('6fb31558cdddef0d1f14cdc872dc179d928214555dd589cc6f235ac06f4706d7') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '848f4b931143ea1181eb9e6ff5bc0805d8abfc6d6199ac7cc3c1da79612dea32', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('848f4b931143ea1181eb9e6ff5bc0805d8abfc6d6199ac7cc3c1da79612dea32') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Args': [], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'LANG=C.UTF-8', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('6fb31558cdddef0d1f14cdc872dc179d928214555dd589cc6f235ac06f4706d7', 'roottestcheckings3blobsparanoid_default') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Args': ['server', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: '--console-address', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: '127.0.0.1:19001', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: '--address', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: ':9001', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: '--certs-dir', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: '/certs', (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: '/data1-1'], (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('848f4b931143ea1181eb9e6ff5bc0805d8abfc6d6199ac7cc3c1da79612dea32', 'roottestcheckings3blobsparanoid_default') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('6fb31558cdddef0d1f14cdc872dc179d928214555dd589cc6f235ac06f4706d7', 'roottestcheckings3blobsparanoid_default', aliases=['resolver', '6fb31558cddd'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('848f4b931143ea1181eb9e6ff5bc0805d8abfc6d6199ac7cc3c1da79612dea32', 'roottestcheckings3blobsparanoid_default', aliases=['minio1', '848f4b931143'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('6fb31558cdddef0d1f14cdc872dc179d928214555dd589cc6f235ac06f4706d7') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('848f4b931143ea1181eb9e6ff5bc0805d8abfc6d6199ac7cc3c1da79612dea32') (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestcheckings3blobsparanoid', service='resolver', number=1) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_resolver_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestcheckings3blobsparanoid', service='minio1', number=1) (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_minio1_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-10-28 23:52:53 [ 499 ] INFO : Trying to connect to Minio... (cluster.py:2965, start) 2026-10-28 23:52:53 [ 499 ] DEBUG : get_instance_ip instance_name=minio1 (cluster.py:2008, get_instance_ip) 2026-10-28 23:52:53 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestcheckings3blobsparanoid_minio1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:53 [ 499 ] DEBUG : get_instance_ip instance_name=proxy1 (cluster.py:2008, get_instance_ip) 2026-10-28 23:52:53 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestcheckings3blobsparanoid_proxy1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:53 [ 499 ] DEBUG : Starting new HTTP connection (1): 172.16.1.5:9001 (connectionpool.py:244, _new_conn) 2026-10-28 23:52:53 [ 499 ] DEBUG : Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-10-28 23:52:53 [ 499 ] WARNING : Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:871, urlopen) 2026-10-28 23:52:53 [ 499 ] DEBUG : Starting new HTTP connection (2): 172.16.1.5:9001 (connectionpool.py:244, _new_conn) 2026-10-28 23:52:53 [ 499 ] DEBUG : Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-10-28 23:52:53 [ 499 ] WARNING : Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:871, urlopen) 2026-10-28 23:52:53 [ 499 ] DEBUG : Starting new HTTP connection (3): 172.16.1.5:9001 (connectionpool.py:244, _new_conn) 2026-10-28 23:52:53 [ 499 ] DEBUG : Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-10-28 23:52:53 [ 499 ] WARNING : Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:871, urlopen) 2026-10-28 23:52:53 [ 499 ] DEBUG : Starting new HTTP connection (4): 172.16.1.5:9001 (connectionpool.py:244, _new_conn) 2026-10-28 23:52:53 [ 499 ] DEBUG : Can't connect to Minio: HTTPConnectionPool(host='172.16.1.5', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2542, wait_minio_to_start) 2026-10-28 23:52:54 [ 499 ] DEBUG : Starting new HTTP connection (5): 172.16.1.5:9001 (connectionpool.py:244, _new_conn) 2026-10-28 23:52:54 [ 499 ] DEBUG : http://172.16.1.5:9001 "GET / HTTP/1.1" 200 0 (connectionpool.py:546, _make_request) 2026-10-28 23:52:54 [ 499 ] DEBUG : Connected to Minio. (cluster.py:2522, wait_minio_to_start) 2026-10-28 23:52:54 [ 499 ] DEBUG : http://172.16.1.5:9001 "GET /root?location= HTTP/1.1" 404 0 (connectionpool.py:546, _make_request) 2026-10-28 23:52:54 [ 499 ] DEBUG : http://172.16.1.5:9001 "PUT /root HTTP/1.1" 200 0 (connectionpool.py:546, _make_request) 2026-10-28 23:52:54 [ 499 ] DEBUG : S3 bucket 'root' created (cluster.py:2537, wait_minio_to_start) 2026-10-28 23:52:54 [ 499 ] DEBUG : http://172.16.1.5:9001 "GET /root2?location= HTTP/1.1" 404 0 (connectionpool.py:546, _make_request) 2026-10-28 23:52:54 [ 499 ] DEBUG : http://172.16.1.5:9001 "PUT /root2 HTTP/1.1" 200 0 (connectionpool.py:546, _make_request) 2026-10-28 23:52:54 [ 499 ] DEBUG : S3 bucket 'root2' created (cluster.py:2537, wait_minio_to_start) 2026-10-28 23:52:54 [ 499 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env --project-name roottestcheckings3blobsparanoid --file /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node_with_inf_s3_retries/docker-compose.yml up -d --no-recreate') (cluster.py:3002, start) 2026-10-28 23:52:54 [ 499 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env', '--project-name', 'roottestcheckings3blobsparanoid', '--file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node_with_inf_s3_retries/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:113, run_and_check) 2026-10-28 23:52:55 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_node_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:52:55 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_node_with_inf_s3_retries_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:52:55 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_node_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:55 [ 499 ] DEBUG : Stderr:Creating roottestcheckings3blobsparanoid_node_with_inf_s3_retries_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:52:55 [ 499 ] DEBUG : ClickHouse instance created (cluster.py:3010, start) 2026-10-28 23:52:55 [ 499 ] DEBUG : get_instance_ip instance_name=node (cluster.py:2008, get_instance_ip) 2026-10-28 23:52:55 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestcheckings3blobsparanoid_node_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:55 [ 499 ] DEBUG : Waiting for ClickHouse start in node, ip: 172.16.1.6... (cluster.py:3017, start) 2026-10-28 23:52:55 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestcheckings3blobsparanoid_node_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:55 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:55 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:55 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:55 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:55 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:55 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:55 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:55 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:56 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:56 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:56 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:56 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:56 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:56 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:56 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:56 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:56 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:56 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/4ed2352f6bf291ee66b88c769c206e807830bf1c424ebbf0c1b8ef963b2576cd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : ClickHouse node started (cluster.py:3021, start) 2026-10-28 23:52:57 [ 499 ] DEBUG : get_instance_ip instance_name=node_with_inf_s3_retries (cluster.py:2008, get_instance_ip) 2026-10-28 23:52:57 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestcheckings3blobsparanoid_node_with_inf_s3_retries_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : Waiting for ClickHouse start in node_with_inf_s3_retries, ip: 172.16.1.7... (cluster.py:3017, start) 2026-10-28 23:52:57 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestcheckings3blobsparanoid_node_with_inf_s3_retries_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/a486f539c04422d61ab34b1bb67a680275612c9bc7ef1d78f5985a23a0bd244c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : http://localhost:None "GET /v1.42/containers/a486f539c04422d61ab34b1bb67a680275612c9bc7ef1d78f5985a23a0bd244c/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2026-10-28 23:52:57 [ 499 ] DEBUG : ClickHouse node_with_inf_s3_retries started (cluster.py:3021, start) 2026-10-28 23:52:57 [ 499 ] INFO : Cluster started (test.py:39, cluster) 2026-10-28 23:52:57 [ 499 ] INFO : Starting mock server broken_s3.py (mock_servers.py:18, start_mock_servers) 2026-10-28 23:52:57 [ 499 ] DEBUG : run container_id:roottestcheckings3blobsparanoid_resolver_1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname broken_s3.py) && echo aW1wb3J0IGxvZ2dpbmcKaW1wb3J0IHN5cwppbXBvcnQgdGhyZWFkaW5nCmltcG9ydCByYW5kb20KaW1wb3J0IHRpbWUKaW1wb3J0IHVybGxpYi5wYXJzZQppbXBvcnQgaHR0cC5zZXJ2ZXIKaW1wb3J0IHNvY2tldHNlcnZlcgppbXBvcnQgc3RyaW5nCmltcG9ydCBzb2NrZXQKaW1wb3J0IHN0cnVjdAoKCklORl9DT1VOVCA9IDEwMDAwMDAwMAoKCmRlZiBfYW5kX3RoZW4odmFsdWUsIGZ1bmMpOgogICAgYXNzZXJ0IGNhbGxhYmxlKGZ1bmMpCiAgICByZXR1cm4gTm9uZSBpZiB2YWx1ZSBpcyBOb25lIGVsc2UgZnVuYyh2YWx1ZSkKCgpjbGFzcyBNb2NrQ29udHJvbDoKICAgIGRlZiBfX2luaXRfXyhzZWxmLCBjbHVzdGVyLCBjb250YWluZXIsIHBvcnQpOgogICAgICAgIHNlbGYuX2NsdXN0ZXIgPSBjbHVzdGVyCiAgICAgICAgc2VsZi5fY29udGFpbmVyID0gY29udGFpbmVyCiAgICAgICAgc2VsZi5fcG9ydCA9IHBvcnQKCiAgICBkZWYgcmVzZXQoc2VsZik6CiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWwogICAgICAgICAgICAgICAgImN1cmwiLAogICAgICAgICAgICAgICAgIi1zIiwKICAgICAgICAgICAgICAgIGYiaHR0cDovL2xvY2FsaG9zdDp7c2VsZi5fcG9ydH0vbW9ja19zZXR0aW5ncy9yZXNldCIsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX2FjdGlvbihzZWxmLCB3aGVuLCBjb3VudD1Ob25lLCBhZnRlcj1Ob25lLCBhY3Rpb249Tm9uZSwgYWN0aW9uX2FyZ3M9Tm9uZSk6CiAgICAgICAgdXJsID0gZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL3t3aGVufT9ub3RoaW5nPTEiCgogICAgICAgIGlmIGNvdW50IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImY291bnQ9e2NvdW50fSIKCiAgICAgICAgaWYgYWZ0ZXIgaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZhZnRlcj17YWZ0ZXJ9IgoKICAgICAgICBpZiBhY3Rpb24gaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZhY3Rpb249e2FjdGlvbn0iCgogICAgICAgIGlmIGFjdGlvbl9hcmdzIGlzIG5vdCBOb25lOgogICAgICAgICAgICBmb3IgeCBpbiBhY3Rpb25fYXJnczoKICAgICAgICAgICAgICAgIHVybCArPSBmIiZhY3Rpb25fYXJncz17eH0iCgogICAgICAgIHJlc3BvbnNlID0gc2VsZi5fY2x1c3Rlci5leGVjX2luX2NvbnRhaW5lcigKICAgICAgICAgICAgc2VsZi5fY2x1c3Rlci5nZXRfY29udGFpbmVyX2lkKHNlbGYuX2NvbnRhaW5lciksCiAgICAgICAgICAgIFsKICAgICAgICAgICAgICAgICJjdXJsIiwKICAgICAgICAgICAgICAgICItcyIsCiAgICAgICAgICAgICAgICB1cmwsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX2F0X29iamVjdF91cGxvYWQoc2VsZiwgKiprd2FyZ3MpOgogICAgICAgIHNlbGYuc2V0dXBfYWN0aW9uKCJhdF9vYmplY3RfdXBsb2FkIiwgKiprd2FyZ3MpCgogICAgZGVmIHNldHVwX2F0X3BhcnRfdXBsb2FkKHNlbGYsICoqa3dhcmdzKToKICAgICAgICBzZWxmLnNldHVwX2FjdGlvbigiYXRfcGFydF91cGxvYWQiLCAqKmt3YXJncykKCiAgICBkZWYgc2V0dXBfYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkKHNlbGYsICoqa3dhcmdzKToKICAgICAgICBzZWxmLnNldHVwX2FjdGlvbigiYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIiwgKiprd2FyZ3MpCgogICAgZGVmIHNldHVwX2Zha2VfcHV0cyhzZWxmLCBwYXJ0X2xlbmd0aCk6CiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWwogICAgICAgICAgICAgICAgImN1cmwiLAogICAgICAgICAgICAgICAgIi1zIiwKICAgICAgICAgICAgICAgIGYiaHR0cDovL2xvY2FsaG9zdDp7c2VsZi5fcG9ydH0vbW9ja19zZXR0aW5ncy9mYWtlX3B1dHM/d2hlbl9sZW5ndGhfYmlnZ2VyPXtwYXJ0X2xlbmd0aH0iLAogICAgICAgICAgICBdLAogICAgICAgICAgICBub3Rocm93PVRydWUsCiAgICAgICAgKQogICAgICAgIGFzc2VydCByZXNwb25zZSA9PSAiT0siLCByZXNwb25zZQoKICAgIGRlZiBzZXR1cF9mYWtlX211bHRwYXJ0dXBsb2FkcyhzZWxmKToKICAgICAgICByZXNwb25zZSA9IHNlbGYuX2NsdXN0ZXIuZXhlY19pbl9jb250YWluZXIoCiAgICAgICAgICAgIHNlbGYuX2NsdXN0ZXIuZ2V0X2NvbnRhaW5lcl9pZChzZWxmLl9jb250YWluZXIpLAogICAgICAgICAgICBbCiAgICAgICAgICAgICAgICAiY3VybCIsCiAgICAgICAgICAgICAgICAiLXMiLAogICAgICAgICAgICAgICAgZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL3NldHVwX2Zha2VfbXVsdHBhcnR1cGxvYWRzPyIsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX3Nsb3dfYW5zd2VycygKICAgICAgICBzZWxmLCBtaW5pbWFsX2xlbmd0aD0wLCB0aW1lb3V0PU5vbmUsIHByb2JhYmlsaXR5PU5vbmUsIGNvdW50PU5vbmUKICAgICk6CiAgICAgICAgdXJsID0gKAogICAgICAgICAgICBmImh0dHA6Ly9sb2NhbGhvc3Q6e3NlbGYuX3BvcnR9LyIKICAgICAgICAgICAgZiJtb2NrX3NldHRpbmdzL3Nsb3dfcHV0IgogICAgICAgICAgICBmIj9taW5pbWFsX2xlbmd0aD17bWluaW1hbF9sZW5ndGh9IgogICAgICAgICkKCiAgICAgICAgaWYgdGltZW91dCBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJnRpbWVvdXQ9e3RpbWVvdXR9IgoKICAgICAgICBpZiBwcm9iYWJpbGl0eSBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJnByb2JhYmlsaXR5PXtwcm9iYWJpbGl0eX0iCgogICAgICAgIGlmIGNvdW50IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImY291bnQ9e2NvdW50fSIKCiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWyJjdXJsIiwgIi1zIiwgdXJsXSwKICAgICAgICAgICAgbm90aHJvdz1UcnVlLAogICAgICAgICkKICAgICAgICBhc3NlcnQgcmVzcG9uc2UgPT0gIk9LIiwgcmVzcG9uc2UKCgpjbGFzcyBfU2VydmVyUnVudGltZToKICAgIGNsYXNzIFNsb3dQdXQ6CiAgICAgICAgZGVmIF9faW5pdF9fKAogICAgICAgICAgICBzZWxmLAogICAgICAgICAgICBsb2NrLAogICAgICAgICAgICBwcm9iYWJpbGl0eV89Tm9uZSwKICAgICAgICAgICAgdGltZW91dF89Tm9uZSwKICAgICAgICAgICAgbWluaW1hbF9sZW5ndGhfPU5vbmUsCiAgICAgICAgICAgIGNvdW50Xz1Ob25lLAogICAgICAgICk6CiAgICAgICAgICAgIHNlbGYubG9jayA9IGxvY2sKICAgICAgICAgICAgc2VsZi5wcm9iYWJpbGl0eSA9IHByb2JhYmlsaXR5XyBpZiBwcm9iYWJpbGl0eV8gaXMgbm90IE5vbmUgZWxzZSAxCiAgICAgICAgICAgIHNlbGYudGltZW91dCA9IHRpbWVvdXRfIGlmIHRpbWVvdXRfIGlzIG5vdCBOb25lIGVsc2UgMC4xCiAgICAgICAgICAgIHNlbGYubWluaW1hbF9sZW5ndGggPSBtaW5pbWFsX2xlbmd0aF8gaWYgbWluaW1hbF9sZW5ndGhfIGlzIG5vdCBOb25lIGVsc2UgMAogICAgICAgICAgICBzZWxmLmNvdW50ID0gY291bnRfIGlmIGNvdW50XyBpcyBub3QgTm9uZSBlbHNlIElORl9DT1VOVAoKICAgICAgICBkZWYgX19zdHJfXyhzZWxmKToKICAgICAgICAgICAgcmV0dXJuICgKICAgICAgICAgICAgICAgIGYicHJvYmFiaWxpdHk6e3NlbGYucHJvYmFiaWxpdHl9IgogICAgICAgICAgICAgICAgZiIgdGltZW91dDp7c2VsZi50aW1lb3V0fSIKICAgICAgICAgICAgICAgIGYiIG1pbmltYWxfbGVuZ3RoOntzZWxmLm1pbmltYWxfbGVuZ3RofSIKICAgICAgICAgICAgICAgIGYiIGNvdW50OntzZWxmLmNvdW50fSIKICAgICAgICAgICAgKQoKICAgICAgICBkZWYgZ2V0X3RpbWVvdXQoc2VsZiwgY29udGVudF9sZW5ndGgpOgogICAgICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgICAgIGlmIGNvbnRlbnRfbGVuZ3RoID4gc2VsZi5taW5pbWFsX2xlbmd0aDoKICAgICAgICAgICAgICAgICAgICBpZiBzZWxmLmNvdW50ID4gMDoKICAgICAgICAgICAgICAgICAgICAgICAgaWYgKAogICAgICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuc2xvd19wdXQucHJvYmFiaWxpdHkgPT0gMQogICAgICAgICAgICAgICAgICAgICAgICAgICAgb3IgcmFuZG9tLnJhbmRvbSgpIDw9IF9ydW50aW1lLnNsb3dfcHV0LnByb2JhYmlsaXR5CiAgICAgICAgICAgICAgICAgICAgICAgICk6CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBzZWxmLmNvdW50IC09IDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5zbG93X3B1dC50aW1lb3V0CiAgICAgICAgICAgIHJldHVybiBOb25lCgogICAgY2xhc3MgRXhwZWN0ZWQ1MDBFcnJvckFjdGlvbjoKICAgICAgICBkZWYgaW5qZWN0X2Vycm9yKHNlbGYsIHJlcXVlc3RfaGFuZGxlcik6CiAgICAgICAgICAgIGRhdGEgPSAoCiAgICAgICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4nCiAgICAgICAgICAgICAgICAiPEVycm9yPiIKICAgICAgICAgICAgICAgICI8Q29kZT5FeHBlY3RlZEVycm9yPC9Db2RlPiIKICAgICAgICAgICAgICAgICI8TWVzc2FnZT5tb2NrIHMzIGluamVjdGVkIGVycm9yPC9NZXNzYWdlPiIKICAgICAgICAgICAgICAgICI8UmVxdWVzdElkPnR4ZmJkNTY2ZDAzMDQyNDc0ODg4MTkzLTAwNjA4ZDc1Mzc8L1JlcXVlc3RJZD4iCiAgICAgICAgICAgICAgICAiPC9FcnJvcj4iCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndyaXRlX2Vycm9yKGRhdGEpCgogICAgY2xhc3MgUmVkaXJlY3RBY3Rpb246CiAgICAgICAgZGVmIF9faW5pdF9fKHNlbGYsIGhvc3Q9ImxvY2FsaG9zdCIsIHBvcnQ9MSk6CiAgICAgICAgICAgIHNlbGYuZHN0X2hvc3QgPSBfYW5kX3RoZW4oaG9zdCwgc3RyKQogICAgICAgICAgICBzZWxmLmRzdF9wb3J0ID0gX2FuZF90aGVuKHBvcnQsIGludCkKCiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIucmVkaXJlY3QoaG9zdD1zZWxmLmRzdF9ob3N0LCBwb3J0PXNlbGYuZHN0X3BvcnQpCgogICAgY2xhc3MgQ29ubmVjdGlvblJlc2V0QnlQZWVyQWN0aW9uOgogICAgICAgIGRlZiBfX2luaXRfXyhzZWxmLCB3aXRoX3BhcnRpYWxfZGF0YT1Ob25lKToKICAgICAgICAgICAgc2VsZi5wYXJ0aWFsX2RhdGEgPSAiIgogICAgICAgICAgICBpZiB3aXRoX3BhcnRpYWxfZGF0YSBpcyBub3QgTm9uZSBhbmQgd2l0aF9wYXJ0aWFsX2RhdGEgPT0gIjEiOgogICAgICAgICAgICAgICAgc2VsZi5wYXJ0aWFsX2RhdGEgPSAoCiAgICAgICAgICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICAgICAgICAgIjxJbml0aWF0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD5cbiIKICAgICAgICAgICAgICAgICkKCiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIucmVhZF9hbGxfaW5wdXQoKQoKICAgICAgICAgICAgaWYgc2VsZi5wYXJ0aWFsX2RhdGE6CiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgMTAwMDApCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuZW5kX2hlYWRlcnMoKQogICAgICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndmaWxlLndyaXRlKGJ5dGVzKHNlbGYucGFydGlhbF9kYXRhLCAiVVRGLTgiKSkKCiAgICAgICAgICAgIHRpbWUuc2xlZXAoMSkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLmNvbm5lY3Rpb24uc2V0c29ja29wdCgKICAgICAgICAgICAgICAgIHNvY2tldC5TT0xfU09DS0VULCBzb2NrZXQuU09fTElOR0VSLCBzdHJ1Y3QucGFjaygiaWkiLCAxLCAwKQogICAgICAgICAgICApCiAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5jb25uZWN0aW9uLmNsb3NlKCkKCiAgICBjbGFzcyBCcm9rZW5QaXBlQWN0aW9uOgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgIyBwYXJ0aWFsIHJlYWQKICAgICAgICAgICAgc2VsZi5yZmlsZS5yZWFkKDUwKQoKICAgICAgICAgICAgdGltZS5zbGVlcCgxKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuY29ubmVjdGlvbi5zZXRzb2Nrb3B0KAogICAgICAgICAgICAgICAgc29ja2V0LlNPTF9TT0NLRVQsIHNvY2tldC5TT19MSU5HRVIsIHN0cnVjdC5wYWNrKCJpaSIsIDEsIDApCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLmNvbm5lY3Rpb24uY2xvc2UoKQoKICAgIGNsYXNzIENvbm5lY3Rpb25SZWZ1c2VkQWN0aW9uKFJlZGlyZWN0QWN0aW9uKToKICAgICAgICBwYXNzCgogICAgY2xhc3MgQ291bnRBZnRlcjoKICAgICAgICBkZWYgX19pbml0X18oCiAgICAgICAgICAgIHNlbGYsIGxvY2ssIGNvdW50Xz1Ob25lLCBhZnRlcl89Tm9uZSwgYWN0aW9uXz1Ob25lLCBhY3Rpb25fYXJnc189W10KICAgICAgICApOgogICAgICAgICAgICBzZWxmLmxvY2sgPSBsb2NrCgogICAgICAgICAgICBzZWxmLmNvdW50ID0gY291bnRfIGlmIGNvdW50XyBpcyBub3QgTm9uZSBlbHNlIElORl9DT1VOVAogICAgICAgICAgICBzZWxmLmFmdGVyID0gYWZ0ZXJfIGlmIGFmdGVyXyBpcyBub3QgTm9uZSBlbHNlIDAKICAgICAgICAgICAgc2VsZi5hY3Rpb24gPSBhY3Rpb25fCiAgICAgICAgICAgIHNlbGYuYWN0aW9uX2FyZ3MgPSBhY3Rpb25fYXJnc18KCiAgICAgICAgICAgIGlmIHNlbGYuYWN0aW9uID09ICJjb25uZWN0aW9uX3JlZnVzZWQiOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuQ29ubmVjdGlvblJlZnVzZWRBY3Rpb24oKQogICAgICAgICAgICBlbGlmIHNlbGYuYWN0aW9uID09ICJjb25uZWN0aW9uX3Jlc2V0X2J5X3BlZXIiOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuQ29ubmVjdGlvblJlc2V0QnlQZWVyQWN0aW9uKAogICAgICAgICAgICAgICAgICAgICpzZWxmLmFjdGlvbl9hcmdzCiAgICAgICAgICAgICAgICApCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gImJyb2tlbl9waXBlIjoKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLkJyb2tlblBpcGVBY3Rpb24oKQogICAgICAgICAgICBlbGlmIHNlbGYuYWN0aW9uID09ICJyZWRpcmVjdF90byI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5SZWRpcmVjdEFjdGlvbigqc2VsZi5hY3Rpb25fYXJncykKICAgICAgICAgICAgZWxzZToKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLkV4cGVjdGVkNTAwRXJyb3JBY3Rpb24oKQoKICAgICAgICBAc3RhdGljbWV0aG9kCiAgICAgICAgZGVmIGZyb21fY2dpX3BhcmFtcyhsb2NrLCBwYXJhbXMpOgogICAgICAgICAgICByZXR1cm4gX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlcigKICAgICAgICAgICAgICAgIGxvY2s9bG9jaywKICAgICAgICAgICAgICAgIGNvdW50Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgiY291bnQiLCBbTm9uZV0pWzBdLCBpbnQpLAogICAgICAgICAgICAgICAgYWZ0ZXJfPV9hbmRfdGhlbihwYXJhbXMuZ2V0KCJhZnRlciIsIFtOb25lXSlbMF0sIGludCksCiAgICAgICAgICAgICAgICBhY3Rpb25fPXBhcmFtcy5nZXQoImFjdGlvbiIsIFtOb25lXSlbMF0sCiAgICAgICAgICAgICAgICBhY3Rpb25fYXJnc189cGFyYW1zLmdldCgiYWN0aW9uX2FyZ3MiLCBbXSksCiAgICAgICAgICAgICkKCiAgICAgICAgZGVmIF9fc3RyX18oc2VsZik6CiAgICAgICAgICAgIHJldHVybiBmImNvdW50OntzZWxmLmNvdW50fSBhZnRlcjp7c2VsZi5hZnRlcn0gYWN0aW9uOntzZWxmLmFjdGlvbn0gYWN0aW9uX2FyZ3M6e3NlbGYuYWN0aW9uX2FyZ3N9IgoKICAgICAgICBkZWYgaGFzX2VmZmVjdChzZWxmKToKICAgICAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgICAgICBpZiBzZWxmLmFmdGVyOgogICAgICAgICAgICAgICAgICAgIHNlbGYuYWZ0ZXIgLT0gMQogICAgICAgICAgICAgICAgaWYgc2VsZi5hZnRlciA9PSAwOgogICAgICAgICAgICAgICAgICAgIGlmIHNlbGYuY291bnQ6CiAgICAgICAgICAgICAgICAgICAgICAgIHNlbGYuY291bnQgLT0gMQogICAgICAgICAgICAgICAgICAgICAgICByZXR1cm4gVHJ1ZQogICAgICAgICAgICAgICAgcmV0dXJuIEZhbHNlCgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyLmluamVjdF9lcnJvcihyZXF1ZXN0X2hhbmRsZXIpCgogICAgZGVmIF9faW5pdF9fKHNlbGYpOgogICAgICAgIHNlbGYubG9jayA9IHRocmVhZGluZy5Mb2NrKCkKICAgICAgICBzZWxmLmF0X3BhcnRfdXBsb2FkID0gTm9uZQogICAgICAgIHNlbGYuYXRfb2JqZWN0X3VwbG9hZCA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfdXBsb2FkcyA9IGRpY3QoKQogICAgICAgIHNlbGYuc2xvd19wdXQgPSBOb25lCiAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgc2VsZi5hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQgPSBOb25lCgogICAgZGVmIHJlZ2lzdGVyX2Zha2VfdXBsb2FkKHNlbGYsIHVwbG9hZF9pZCwga2V5KToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHNbdXBsb2FkX2lkXSA9IGtleQoKICAgIGRlZiBpc19mYWtlX3VwbG9hZChzZWxmLCB1cGxvYWRfaWQsIGtleSk6CiAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgIGlmIHVwbG9hZF9pZCBpbiBzZWxmLmZha2VfdXBsb2FkczoKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLmZha2VfdXBsb2Fkc1t1cGxvYWRfaWRdID09IGtleQogICAgICAgIHJldHVybiBGYWxzZQoKICAgIGRlZiByZXNldChzZWxmKToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5hdF9wYXJ0X3VwbG9hZCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5hdF9vYmplY3RfdXBsb2FkID0gTm9uZQogICAgICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHMgPSBkaWN0KCkKICAgICAgICAgICAgc2VsZi5zbG93X3B1dCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgICAgIHNlbGYuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkID0gTm9uZQoKCl9ydW50aW1lID0gX1NlcnZlclJ1bnRpbWUoKQoKCmRlZiBnZXRfcmFuZG9tX3N0cmluZyhsZW5ndGgpOgogICAgIyBjaG9vc2UgZnJvbSBhbGwgbG93ZXJjYXNlIGxldHRlcgogICAgbGV0dGVycyA9IHN0cmluZy5hc2NpaV9sb3dlcmNhc2UKICAgIHJlc3VsdF9zdHIgPSAiIi5qb2luKHJhbmRvbS5jaG9pY2UobGV0dGVycykgZm9yIGkgaW4gcmFuZ2UobGVuZ3RoKSkKICAgIHJldHVybiByZXN1bHRfc3RyCgoKY2xhc3MgUmVxdWVzdEhhbmRsZXIoaHR0cC5zZXJ2ZXIuQmFzZUhUVFBSZXF1ZXN0SGFuZGxlcik6CiAgICBkZWYgX29rKHNlbGYpOgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYiJPSyIpCgogICAgZGVmIF9waW5nKHNlbGYpOgogICAgICAgIHNlbGYuX29rKCkKCiAgICBkZWYgcmVhZF9hbGxfaW5wdXQoc2VsZik6CiAgICAgICAgY29udGVudF9sZW5ndGggPSBpbnQoc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiLCAwKSkKICAgICAgICB0b19yZWFkID0gY29udGVudF9sZW5ndGgKICAgICAgICB3aGlsZSB0b19yZWFkID4gMDoKICAgICAgICAgICAgIyByZWFkIGNvbnRlbnQgaW4gb3JkZXIgdG8gYXZvaWQgZXJyb3Igb24gY2xpZW50CiAgICAgICAgICAgICMgUG9jbzo6RXhjZXB0aW9uLiBDb2RlOiAxMDAwLCBlLmNvZGUoKSA9IDMyLCBJL08gZXJyb3I6IEJyb2tlbiBwaXBlCiAgICAgICAgICAgICMgZG8gaXQgcGllY2UgYnkgcGllY2UgaW4gb3JkZXIgdG8gYXZvaWQgYmlnIGFsbG9jYXRpb24KICAgICAgICAgICAgc2l6ZSA9IG1pbih0b19yZWFkLCAxMDI0KQogICAgICAgICAgICBzdHIoc2VsZi5yZmlsZS5yZWFkKHNpemUpKQogICAgICAgICAgICB0b19yZWFkIC09IHNpemUKCiAgICBkZWYgcmVkaXJlY3Qoc2VsZiwgaG9zdD1Ob25lLCBwb3J0PU5vbmUpOgogICAgICAgIGlmIGhvc3QgaXMgTm9uZSBhbmQgcG9ydCBpcyBOb25lOgogICAgICAgICAgICBob3N0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1faG9zdAogICAgICAgICAgICBwb3J0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1fcG9ydAoKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKCiAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDMwNykKICAgICAgICB1cmwgPSBmImh0dHA6Ly97aG9zdH06e3BvcnR9e3NlbGYucGF0aH0iCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgicmVkaXJlY3QgdG8gJXMiLCB1cmwpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiTG9jYXRpb24iLCB1cmwpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIlJlZGlyZWN0ZWQiKQoKICAgIGRlZiB3cml0ZV9lcnJvcihzZWxmLCBkYXRhLCBjb250ZW50X2xlbmd0aD1Ob25lKToKICAgICAgICBpZiBjb250ZW50X2xlbmd0aCBpcyBOb25lOgogICAgICAgICAgICBjb250ZW50X2xlbmd0aCA9IGxlbihkYXRhKQogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoIndyaXRlX2Vycm9yICVzIiwgZGF0YSkKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNTAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBzdHIoY29udGVudF9sZW5ndGgpKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIGlmIGRhdGE6CiAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9mYWtlX3B1dF9vayhzZWxmKToKICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJmYWtlIHB1dCIpCgogICAgICAgIHNlbGYucmVhZF9hbGxfaW5wdXQoKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiRVRhZyIsICJiNTQzNTdmYWYwNjMyY2NlNDZlOTQyZmE2ODM1NmIzOCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCAwKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgIGRlZiBfZmFrZV91cGxvYWRzKHNlbGYsIHBhdGgsIHVwbG9hZF9pZCk6CiAgICAgICAgc2VsZi5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgIHBhcnRzID0gW3ggZm9yIHggaW4gcGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYnVja2V0ID0gcGFydHNbMF0KICAgICAgICBrZXkgPSAiLyIuam9pbihwYXJ0c1sxOl0pCiAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICI8SW5pdGlhdGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgICAgIGYiPEJ1Y2tldD57YnVja2V0fTwvQnVja2V0PiIKICAgICAgICAgICAgZiI8S2V5PntrZXl9PC9LZXk+IgogICAgICAgICAgICBmIjxVcGxvYWRJZD57dXBsb2FkX2lkfTwvVXBsb2FkSWQ+IgogICAgICAgICAgICAiPC9Jbml0aWF0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD4iCiAgICAgICAgKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4oZGF0YSkpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9mYWtlX3Bvc3Rfb2soc2VsZiwgcGF0aCk6CiAgICAgICAgc2VsZi5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgIHBhcnRzID0gW3ggZm9yIHggaW4gcGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYnVja2V0ID0gcGFydHNbMF0KICAgICAgICBrZXkgPSAiLyIuam9pbihwYXJ0c1sxOl0pCiAgICAgICAgbG9jYXRpb24gPSAiaHR0cDovL0V4YW1wbGUtQnVja2V0LnMzLlJlZ2lvbi5hbWF6b25hd3MuY29tLyIgKyBwYXRoCiAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICI8Q29tcGxldGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgICAgIGYiPExvY2F0aW9uPntsb2NhdGlvbn08L0xvY2F0aW9uPlxuIgogICAgICAgICAgICBmIjxCdWNrZXQ+e2J1Y2tldH08L0J1Y2tldD5cbiIKICAgICAgICAgICAgZiI8S2V5PntrZXl9PC9LZXk+XG4iCiAgICAgICAgICAgIGYnPEVUYWc+IjM4NThmNjIyMzBhYzNjOTE1ZjMwMGM2NjQzMTJjMTFmLTkiPC9FVGFnPlxuJwogICAgICAgICAgICBmIjwvQ29tcGxldGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4oZGF0YSkpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9tb2NrX3NldHRpbmdzKHNlbGYpOgogICAgICAgIHBhcnRzID0gdXJsbGliLnBhcnNlLnVybHNwbGl0KHNlbGYucGF0aCkKICAgICAgICBwYXRoID0gW3ggZm9yIHggaW4gcGFydHMucGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYXNzZXJ0IHBhdGhbMF0gPT0gIm1vY2tfc2V0dGluZ3MiLCBwYXRoCiAgICAgICAgaWYgbGVuKHBhdGgpIDwgMjoKICAgICAgICAgICAgcmV0dXJuIHNlbGYud3JpdGVfZXJyb3IoIl9tb2NrX3NldHRpbmdzOiB3cm9uZyBjb21tYW5kIikKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAiYXRfcGFydF91cGxvYWQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCA9IF9TZXJ2ZXJSdW50aW1lLkNvdW50QWZ0ZXIuZnJvbV9jZ2lfcGFyYW1zKAogICAgICAgICAgICAgICAgX3J1bnRpbWUubG9jaywgcGFyYW1zCiAgICAgICAgICAgICkKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IGF0X3BhcnRfdXBsb2FkICVzIiwgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImF0X29iamVjdF91cGxvYWQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkID0gX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlci5mcm9tX2NnaV9wYXJhbXMoCiAgICAgICAgICAgICAgICBfcnVudGltZS5sb2NrLCBwYXJhbXMKICAgICAgICAgICAgKQogICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJzZXQgYXRfb2JqZWN0X3VwbG9hZCAlcyIsIF9ydW50aW1lLmF0X29iamVjdF91cGxvYWQpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImZha2VfcHV0cyI6CiAgICAgICAgICAgIHBhcmFtcyA9IHVybGxpYi5wYXJzZS5wYXJzZV9xcyhwYXJ0cy5xdWVyeSwga2VlcF9ibGFua192YWx1ZXM9RmFsc2UpCiAgICAgICAgICAgIF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IGludCgKICAgICAgICAgICAgICAgIHBhcmFtcy5nZXQoIndoZW5fbGVuZ3RoX2JpZ2dlciIsIFsxMDI0ICogMTAyNF0pWzBdCiAgICAgICAgICAgICkKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IGZha2VfcHV0cyAlcyIsIF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlcikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAic2xvd19wdXQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5zbG93X3B1dCA9IF9TZXJ2ZXJSdW50aW1lLlNsb3dQdXQoCiAgICAgICAgICAgICAgICBsb2NrPV9ydW50aW1lLmxvY2ssCiAgICAgICAgICAgICAgICBtaW5pbWFsX2xlbmd0aF89X2FuZF90aGVuKHBhcmFtcy5nZXQoIm1pbmltYWxfbGVuZ3RoIiwgW05vbmVdKVswXSwgaW50KSwKICAgICAgICAgICAgICAgIHByb2JhYmlsaXR5Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgicHJvYmFiaWxpdHkiLCBbTm9uZV0pWzBdLCBmbG9hdCksCiAgICAgICAgICAgICAgICB0aW1lb3V0Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgidGltZW91dCIsIFtOb25lXSlbMF0sIGZsb2F0KSwKICAgICAgICAgICAgICAgIGNvdW50Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgiY291bnQiLCBbTm9uZV0pWzBdLCBpbnQpLAogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInNldCBzbG93IHB1dCAlcyIsIF9ydW50aW1lLnNsb3dfcHV0KQogICAgICAgICAgICByZXR1cm4gc2VsZi5fb2soKQoKICAgICAgICBpZiBwYXRoWzFdID09ICJzZXR1cF9mYWtlX211bHRwYXJ0dXBsb2FkcyI6CiAgICAgICAgICAgIF9ydW50aW1lLmZha2VfbXVsdGlwYXJ0X3VwbG9hZCA9IFRydWUKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IHNldHVwX2Zha2VfbXVsdHBhcnR1cGxvYWRzIikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAiYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIjoKICAgICAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICAgICAgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkID0gKAogICAgICAgICAgICAgICAgX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlci5mcm9tX2NnaV9wYXJhbXMoX3J1bnRpbWUubG9jaywgcGFyYW1zKQogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICAgICAic2V0IGF0X2NyZWF0ZV9tdWx0aV9wYXJ0X3VwbG9hZCAlcyIsCiAgICAgICAgICAgICAgICBfcnVudGltZS5hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQsCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAicmVzZXQiOgogICAgICAgICAgICBfcnVudGltZS5yZXNldCgpCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInJlc2V0IikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgcmV0dXJuIHNlbGYud3JpdGVfZXJyb3IoIl9tb2NrX3NldHRpbmdzOiB3cm9uZyBjb21tYW5kIikKCiAgICBkZWYgZG9fR0VUKHNlbGYpOgogICAgICAgIGlmIHNlbGYucGF0aCA9PSAiLyI6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9waW5nKCkKCiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9tb2NrX3NldHRpbmdzIik6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9tb2NrX3NldHRpbmdzKCkKCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgiZ2V0IHJlZGlyZWN0IikKICAgICAgICByZXR1cm4gc2VsZi5yZWRpcmVjdCgpCgogICAgZGVmIGRvX1BVVChzZWxmKToKICAgICAgICBjb250ZW50X2xlbmd0aCA9IGludChzZWxmLmhlYWRlcnMuZ2V0KCJDb250ZW50LUxlbmd0aCIsIDApKQoKICAgICAgICBpZiBfcnVudGltZS5zbG93X3B1dCBpcyBub3QgTm9uZToKICAgICAgICAgICAgdGltZW91dCA9IF9ydW50aW1lLnNsb3dfcHV0LmdldF90aW1lb3V0KGNvbnRlbnRfbGVuZ3RoKQogICAgICAgICAgICBpZiB0aW1lb3V0IGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2xvdyBwdXQgJXMiLCB0aW1lb3V0KQogICAgICAgICAgICAgICAgdGltZS5zbGVlcCh0aW1lb3V0KQoKICAgICAgICBwYXJ0cyA9IHVybGxpYi5wYXJzZS51cmxzcGxpdChzZWxmLnBhdGgpCiAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KCiAgICAgICAgaWYgdXBsb2FkX2lkIGlzIG5vdCBOb25lOgogICAgICAgICAgICBpZiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCBpcyBub3QgTm9uZToKICAgICAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICAgICAgICAgInB1dCBhdF9wYXJ0X3VwbG9hZCAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICB1cGxvYWRfaWQsCiAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICApCgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQuaGFzX2VmZmVjdCgpOgogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZC5pbmplY3RfZXJyb3Ioc2VsZikKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuaXNfZmFrZV91cGxvYWQodXBsb2FkX2lkLCBwYXJ0cy5wYXRoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wdXRfb2soKQogICAgICAgIGVsc2U6CiAgICAgICAgICAgIGlmIF9ydW50aW1lLmF0X29iamVjdF91cGxvYWQgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGVycm9yX2F0X29iamVjdF91cGxvYWQgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuYXRfb2JqZWN0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQogICAgICAgICAgICBpZiBfcnVudGltZS5mYWtlX3B1dF93aGVuX2xlbmd0aF9iaWdnZXIgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBjb250ZW50X2xlbmd0aCA+IF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlcjoKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuZmFrZV9wdXRfd2hlbl9sZW5ndGhfYmlnZ2VyLAogICAgICAgICAgICAgICAgICAgICAgICBjb250ZW50X2xlbmd0aCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3B1dF9vaygpCgogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICJwdXQgcmVkaXJlY3QgJXMiLAogICAgICAgICAgICBwYXJ0cywKICAgICAgICApCiAgICAgICAgcmV0dXJuIHNlbGYucmVkaXJlY3QoKQoKICAgIGRlZiBkb19QT1NUKHNlbGYpOgogICAgICAgIHBhcnRzID0gdXJsbGliLnBhcnNlLnVybHNwbGl0KHNlbGYucGF0aCkKICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCiAgICAgICAgdXBsb2FkcyA9IHBhcmFtcy5nZXQoInVwbG9hZHMiLCBbTm9uZV0pWzBdCiAgICAgICAgaWYgdXBsb2FkcyBpcyBub3QgTm9uZToKICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQoKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgdXBsb2FkX2lkID0gZ2V0X3JhbmRvbV9zdHJpbmcoNSkKICAgICAgICAgICAgICAgIF9ydW50aW1lLnJlZ2lzdGVyX2Zha2VfdXBsb2FkKHVwbG9hZF9pZCwgcGFydHMucGF0aCkKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3VwbG9hZHMocGFydHMucGF0aCwgdXBsb2FkX2lkKQoKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KICAgICAgICBpZiBfcnVudGltZS5pc19mYWtlX3VwbG9hZCh1cGxvYWRfaWQsIHBhcnRzLnBhdGgpOgogICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wb3N0X29rKHBhcnRzLnBhdGgpCgogICAgICAgIHJldHVybiBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fSEVBRChzZWxmKToKICAgICAgICBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fREVMRVRFKHNlbGYpOgogICAgICAgIHNlbGYucmVkaXJlY3QoKQoKCmNsYXNzIF9UaHJlYWRlZEhUVFBTZXJ2ZXIoc29ja2V0c2VydmVyLlRocmVhZGluZ01peEluLCBodHRwLnNlcnZlci5IVFRQU2VydmVyKToKICAgICIiIkhhbmRsZSByZXF1ZXN0cyBpbiBhIHNlcGFyYXRlIHRocmVhZC4iIiIKCiAgICBkZWYgc2V0X3Vwc3RyZWFtKHNlbGYsIHVwc3RyZWFtX2hvc3QsIHVwc3RyZWFtX3BvcnQpOgogICAgICAgIHNlbGYudXBzdHJlYW1faG9zdCA9IHVwc3RyZWFtX2hvc3QKICAgICAgICBzZWxmLnVwc3RyZWFtX3BvcnQgPSB1cHN0cmVhbV9wb3J0CgoKaWYgX19uYW1lX18gPT0gIl9fbWFpbl9fIjoKICAgIGh0dHBkID0gX1RocmVhZGVkSFRUUFNlcnZlcigoIjAuMC4wLjAiLCBpbnQoc3lzLmFyZ3ZbMV0pKSwgUmVxdWVzdEhhbmRsZXIpCiAgICBpZiBsZW4oc3lzLmFyZ3YpID09IDQ6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKHN5cy5hcmd2WzJdLCBzeXMuYXJndlszXSkKICAgIGVsc2U6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKCJtaW5pbzEiLCA5MDAxKQogICAgaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > broken_s3.py'] (cluster.py:2046, exec_in_container) 2026-10-28 23:52:57 [ 499 ] DEBUG : Command:['docker', 'exec', 'roottestcheckings3blobsparanoid_resolver_1', 'bash', '-c', 'mkdir -p $(dirname broken_s3.py) && echo aW1wb3J0IGxvZ2dpbmcKaW1wb3J0IHN5cwppbXBvcnQgdGhyZWFkaW5nCmltcG9ydCByYW5kb20KaW1wb3J0IHRpbWUKaW1wb3J0IHVybGxpYi5wYXJzZQppbXBvcnQgaHR0cC5zZXJ2ZXIKaW1wb3J0IHNvY2tldHNlcnZlcgppbXBvcnQgc3RyaW5nCmltcG9ydCBzb2NrZXQKaW1wb3J0IHN0cnVjdAoKCklORl9DT1VOVCA9IDEwMDAwMDAwMAoKCmRlZiBfYW5kX3RoZW4odmFsdWUsIGZ1bmMpOgogICAgYXNzZXJ0IGNhbGxhYmxlKGZ1bmMpCiAgICByZXR1cm4gTm9uZSBpZiB2YWx1ZSBpcyBOb25lIGVsc2UgZnVuYyh2YWx1ZSkKCgpjbGFzcyBNb2NrQ29udHJvbDoKICAgIGRlZiBfX2luaXRfXyhzZWxmLCBjbHVzdGVyLCBjb250YWluZXIsIHBvcnQpOgogICAgICAgIHNlbGYuX2NsdXN0ZXIgPSBjbHVzdGVyCiAgICAgICAgc2VsZi5fY29udGFpbmVyID0gY29udGFpbmVyCiAgICAgICAgc2VsZi5fcG9ydCA9IHBvcnQKCiAgICBkZWYgcmVzZXQoc2VsZik6CiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWwogICAgICAgICAgICAgICAgImN1cmwiLAogICAgICAgICAgICAgICAgIi1zIiwKICAgICAgICAgICAgICAgIGYiaHR0cDovL2xvY2FsaG9zdDp7c2VsZi5fcG9ydH0vbW9ja19zZXR0aW5ncy9yZXNldCIsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX2FjdGlvbihzZWxmLCB3aGVuLCBjb3VudD1Ob25lLCBhZnRlcj1Ob25lLCBhY3Rpb249Tm9uZSwgYWN0aW9uX2FyZ3M9Tm9uZSk6CiAgICAgICAgdXJsID0gZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL3t3aGVufT9ub3RoaW5nPTEiCgogICAgICAgIGlmIGNvdW50IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImY291bnQ9e2NvdW50fSIKCiAgICAgICAgaWYgYWZ0ZXIgaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZhZnRlcj17YWZ0ZXJ9IgoKICAgICAgICBpZiBhY3Rpb24gaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZhY3Rpb249e2FjdGlvbn0iCgogICAgICAgIGlmIGFjdGlvbl9hcmdzIGlzIG5vdCBOb25lOgogICAgICAgICAgICBmb3IgeCBpbiBhY3Rpb25fYXJnczoKICAgICAgICAgICAgICAgIHVybCArPSBmIiZhY3Rpb25fYXJncz17eH0iCgogICAgICAgIHJlc3BvbnNlID0gc2VsZi5fY2x1c3Rlci5leGVjX2luX2NvbnRhaW5lcigKICAgICAgICAgICAgc2VsZi5fY2x1c3Rlci5nZXRfY29udGFpbmVyX2lkKHNlbGYuX2NvbnRhaW5lciksCiAgICAgICAgICAgIFsKICAgICAgICAgICAgICAgICJjdXJsIiwKICAgICAgICAgICAgICAgICItcyIsCiAgICAgICAgICAgICAgICB1cmwsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX2F0X29iamVjdF91cGxvYWQoc2VsZiwgKiprd2FyZ3MpOgogICAgICAgIHNlbGYuc2V0dXBfYWN0aW9uKCJhdF9vYmplY3RfdXBsb2FkIiwgKiprd2FyZ3MpCgogICAgZGVmIHNldHVwX2F0X3BhcnRfdXBsb2FkKHNlbGYsICoqa3dhcmdzKToKICAgICAgICBzZWxmLnNldHVwX2FjdGlvbigiYXRfcGFydF91cGxvYWQiLCAqKmt3YXJncykKCiAgICBkZWYgc2V0dXBfYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkKHNlbGYsICoqa3dhcmdzKToKICAgICAgICBzZWxmLnNldHVwX2FjdGlvbigiYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIiwgKiprd2FyZ3MpCgogICAgZGVmIHNldHVwX2Zha2VfcHV0cyhzZWxmLCBwYXJ0X2xlbmd0aCk6CiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWwogICAgICAgICAgICAgICAgImN1cmwiLAogICAgICAgICAgICAgICAgIi1zIiwKICAgICAgICAgICAgICAgIGYiaHR0cDovL2xvY2FsaG9zdDp7c2VsZi5fcG9ydH0vbW9ja19zZXR0aW5ncy9mYWtlX3B1dHM/d2hlbl9sZW5ndGhfYmlnZ2VyPXtwYXJ0X2xlbmd0aH0iLAogICAgICAgICAgICBdLAogICAgICAgICAgICBub3Rocm93PVRydWUsCiAgICAgICAgKQogICAgICAgIGFzc2VydCByZXNwb25zZSA9PSAiT0siLCByZXNwb25zZQoKICAgIGRlZiBzZXR1cF9mYWtlX211bHRwYXJ0dXBsb2FkcyhzZWxmKToKICAgICAgICByZXNwb25zZSA9IHNlbGYuX2NsdXN0ZXIuZXhlY19pbl9jb250YWluZXIoCiAgICAgICAgICAgIHNlbGYuX2NsdXN0ZXIuZ2V0X2NvbnRhaW5lcl9pZChzZWxmLl9jb250YWluZXIpLAogICAgICAgICAgICBbCiAgICAgICAgICAgICAgICAiY3VybCIsCiAgICAgICAgICAgICAgICAiLXMiLAogICAgICAgICAgICAgICAgZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL3NldHVwX2Zha2VfbXVsdHBhcnR1cGxvYWRzPyIsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX3Nsb3dfYW5zd2VycygKICAgICAgICBzZWxmLCBtaW5pbWFsX2xlbmd0aD0wLCB0aW1lb3V0PU5vbmUsIHByb2JhYmlsaXR5PU5vbmUsIGNvdW50PU5vbmUKICAgICk6CiAgICAgICAgdXJsID0gKAogICAgICAgICAgICBmImh0dHA6Ly9sb2NhbGhvc3Q6e3NlbGYuX3BvcnR9LyIKICAgICAgICAgICAgZiJtb2NrX3NldHRpbmdzL3Nsb3dfcHV0IgogICAgICAgICAgICBmIj9taW5pbWFsX2xlbmd0aD17bWluaW1hbF9sZW5ndGh9IgogICAgICAgICkKCiAgICAgICAgaWYgdGltZW91dCBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJnRpbWVvdXQ9e3RpbWVvdXR9IgoKICAgICAgICBpZiBwcm9iYWJpbGl0eSBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJnByb2JhYmlsaXR5PXtwcm9iYWJpbGl0eX0iCgogICAgICAgIGlmIGNvdW50IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImY291bnQ9e2NvdW50fSIKCiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWyJjdXJsIiwgIi1zIiwgdXJsXSwKICAgICAgICAgICAgbm90aHJvdz1UcnVlLAogICAgICAgICkKICAgICAgICBhc3NlcnQgcmVzcG9uc2UgPT0gIk9LIiwgcmVzcG9uc2UKCgpjbGFzcyBfU2VydmVyUnVudGltZToKICAgIGNsYXNzIFNsb3dQdXQ6CiAgICAgICAgZGVmIF9faW5pdF9fKAogICAgICAgICAgICBzZWxmLAogICAgICAgICAgICBsb2NrLAogICAgICAgICAgICBwcm9iYWJpbGl0eV89Tm9uZSwKICAgICAgICAgICAgdGltZW91dF89Tm9uZSwKICAgICAgICAgICAgbWluaW1hbF9sZW5ndGhfPU5vbmUsCiAgICAgICAgICAgIGNvdW50Xz1Ob25lLAogICAgICAgICk6CiAgICAgICAgICAgIHNlbGYubG9jayA9IGxvY2sKICAgICAgICAgICAgc2VsZi5wcm9iYWJpbGl0eSA9IHByb2JhYmlsaXR5XyBpZiBwcm9iYWJpbGl0eV8gaXMgbm90IE5vbmUgZWxzZSAxCiAgICAgICAgICAgIHNlbGYudGltZW91dCA9IHRpbWVvdXRfIGlmIHRpbWVvdXRfIGlzIG5vdCBOb25lIGVsc2UgMC4xCiAgICAgICAgICAgIHNlbGYubWluaW1hbF9sZW5ndGggPSBtaW5pbWFsX2xlbmd0aF8gaWYgbWluaW1hbF9sZW5ndGhfIGlzIG5vdCBOb25lIGVsc2UgMAogICAgICAgICAgICBzZWxmLmNvdW50ID0gY291bnRfIGlmIGNvdW50XyBpcyBub3QgTm9uZSBlbHNlIElORl9DT1VOVAoKICAgICAgICBkZWYgX19zdHJfXyhzZWxmKToKICAgICAgICAgICAgcmV0dXJuICgKICAgICAgICAgICAgICAgIGYicHJvYmFiaWxpdHk6e3NlbGYucHJvYmFiaWxpdHl9IgogICAgICAgICAgICAgICAgZiIgdGltZW91dDp7c2VsZi50aW1lb3V0fSIKICAgICAgICAgICAgICAgIGYiIG1pbmltYWxfbGVuZ3RoOntzZWxmLm1pbmltYWxfbGVuZ3RofSIKICAgICAgICAgICAgICAgIGYiIGNvdW50OntzZWxmLmNvdW50fSIKICAgICAgICAgICAgKQoKICAgICAgICBkZWYgZ2V0X3RpbWVvdXQoc2VsZiwgY29udGVudF9sZW5ndGgpOgogICAgICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgICAgIGlmIGNvbnRlbnRfbGVuZ3RoID4gc2VsZi5taW5pbWFsX2xlbmd0aDoKICAgICAgICAgICAgICAgICAgICBpZiBzZWxmLmNvdW50ID4gMDoKICAgICAgICAgICAgICAgICAgICAgICAgaWYgKAogICAgICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuc2xvd19wdXQucHJvYmFiaWxpdHkgPT0gMQogICAgICAgICAgICAgICAgICAgICAgICAgICAgb3IgcmFuZG9tLnJhbmRvbSgpIDw9IF9ydW50aW1lLnNsb3dfcHV0LnByb2JhYmlsaXR5CiAgICAgICAgICAgICAgICAgICAgICAgICk6CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBzZWxmLmNvdW50IC09IDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5zbG93X3B1dC50aW1lb3V0CiAgICAgICAgICAgIHJldHVybiBOb25lCgogICAgY2xhc3MgRXhwZWN0ZWQ1MDBFcnJvckFjdGlvbjoKICAgICAgICBkZWYgaW5qZWN0X2Vycm9yKHNlbGYsIHJlcXVlc3RfaGFuZGxlcik6CiAgICAgICAgICAgIGRhdGEgPSAoCiAgICAgICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4nCiAgICAgICAgICAgICAgICAiPEVycm9yPiIKICAgICAgICAgICAgICAgICI8Q29kZT5FeHBlY3RlZEVycm9yPC9Db2RlPiIKICAgICAgICAgICAgICAgICI8TWVzc2FnZT5tb2NrIHMzIGluamVjdGVkIGVycm9yPC9NZXNzYWdlPiIKICAgICAgICAgICAgICAgICI8UmVxdWVzdElkPnR4ZmJkNTY2ZDAzMDQyNDc0ODg4MTkzLTAwNjA4ZDc1Mzc8L1JlcXVlc3RJZD4iCiAgICAgICAgICAgICAgICAiPC9FcnJvcj4iCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndyaXRlX2Vycm9yKGRhdGEpCgogICAgY2xhc3MgUmVkaXJlY3RBY3Rpb246CiAgICAgICAgZGVmIF9faW5pdF9fKHNlbGYsIGhvc3Q9ImxvY2FsaG9zdCIsIHBvcnQ9MSk6CiAgICAgICAgICAgIHNlbGYuZHN0X2hvc3QgPSBfYW5kX3RoZW4oaG9zdCwgc3RyKQogICAgICAgICAgICBzZWxmLmRzdF9wb3J0ID0gX2FuZF90aGVuKHBvcnQsIGludCkKCiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIucmVkaXJlY3QoaG9zdD1zZWxmLmRzdF9ob3N0LCBwb3J0PXNlbGYuZHN0X3BvcnQpCgogICAgY2xhc3MgQ29ubmVjdGlvblJlc2V0QnlQZWVyQWN0aW9uOgogICAgICAgIGRlZiBfX2luaXRfXyhzZWxmLCB3aXRoX3BhcnRpYWxfZGF0YT1Ob25lKToKICAgICAgICAgICAgc2VsZi5wYXJ0aWFsX2RhdGEgPSAiIgogICAgICAgICAgICBpZiB3aXRoX3BhcnRpYWxfZGF0YSBpcyBub3QgTm9uZSBhbmQgd2l0aF9wYXJ0aWFsX2RhdGEgPT0gIjEiOgogICAgICAgICAgICAgICAgc2VsZi5wYXJ0aWFsX2RhdGEgPSAoCiAgICAgICAgICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICAgICAgICAgIjxJbml0aWF0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD5cbiIKICAgICAgICAgICAgICAgICkKCiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIucmVhZF9hbGxfaW5wdXQoKQoKICAgICAgICAgICAgaWYgc2VsZi5wYXJ0aWFsX2RhdGE6CiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgMTAwMDApCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuZW5kX2hlYWRlcnMoKQogICAgICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndmaWxlLndyaXRlKGJ5dGVzKHNlbGYucGFydGlhbF9kYXRhLCAiVVRGLTgiKSkKCiAgICAgICAgICAgIHRpbWUuc2xlZXAoMSkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLmNvbm5lY3Rpb24uc2V0c29ja29wdCgKICAgICAgICAgICAgICAgIHNvY2tldC5TT0xfU09DS0VULCBzb2NrZXQuU09fTElOR0VSLCBzdHJ1Y3QucGFjaygiaWkiLCAxLCAwKQogICAgICAgICAgICApCiAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5jb25uZWN0aW9uLmNsb3NlKCkKCiAgICBjbGFzcyBCcm9rZW5QaXBlQWN0aW9uOgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgIyBwYXJ0aWFsIHJlYWQKICAgICAgICAgICAgc2VsZi5yZmlsZS5yZWFkKDUwKQoKICAgICAgICAgICAgdGltZS5zbGVlcCgxKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuY29ubmVjdGlvbi5zZXRzb2Nrb3B0KAogICAgICAgICAgICAgICAgc29ja2V0LlNPTF9TT0NLRVQsIHNvY2tldC5TT19MSU5HRVIsIHN0cnVjdC5wYWNrKCJpaSIsIDEsIDApCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLmNvbm5lY3Rpb24uY2xvc2UoKQoKICAgIGNsYXNzIENvbm5lY3Rpb25SZWZ1c2VkQWN0aW9uKFJlZGlyZWN0QWN0aW9uKToKICAgICAgICBwYXNzCgogICAgY2xhc3MgQ291bnRBZnRlcjoKICAgICAgICBkZWYgX19pbml0X18oCiAgICAgICAgICAgIHNlbGYsIGxvY2ssIGNvdW50Xz1Ob25lLCBhZnRlcl89Tm9uZSwgYWN0aW9uXz1Ob25lLCBhY3Rpb25fYXJnc189W10KICAgICAgICApOgogICAgICAgICAgICBzZWxmLmxvY2sgPSBsb2NrCgogICAgICAgICAgICBzZWxmLmNvdW50ID0gY291bnRfIGlmIGNvdW50XyBpcyBub3QgTm9uZSBlbHNlIElORl9DT1VOVAogICAgICAgICAgICBzZWxmLmFmdGVyID0gYWZ0ZXJfIGlmIGFmdGVyXyBpcyBub3QgTm9uZSBlbHNlIDAKICAgICAgICAgICAgc2VsZi5hY3Rpb24gPSBhY3Rpb25fCiAgICAgICAgICAgIHNlbGYuYWN0aW9uX2FyZ3MgPSBhY3Rpb25fYXJnc18KCiAgICAgICAgICAgIGlmIHNlbGYuYWN0aW9uID09ICJjb25uZWN0aW9uX3JlZnVzZWQiOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuQ29ubmVjdGlvblJlZnVzZWRBY3Rpb24oKQogICAgICAgICAgICBlbGlmIHNlbGYuYWN0aW9uID09ICJjb25uZWN0aW9uX3Jlc2V0X2J5X3BlZXIiOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuQ29ubmVjdGlvblJlc2V0QnlQZWVyQWN0aW9uKAogICAgICAgICAgICAgICAgICAgICpzZWxmLmFjdGlvbl9hcmdzCiAgICAgICAgICAgICAgICApCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gImJyb2tlbl9waXBlIjoKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLkJyb2tlblBpcGVBY3Rpb24oKQogICAgICAgICAgICBlbGlmIHNlbGYuYWN0aW9uID09ICJyZWRpcmVjdF90byI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5SZWRpcmVjdEFjdGlvbigqc2VsZi5hY3Rpb25fYXJncykKICAgICAgICAgICAgZWxzZToKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLkV4cGVjdGVkNTAwRXJyb3JBY3Rpb24oKQoKICAgICAgICBAc3RhdGljbWV0aG9kCiAgICAgICAgZGVmIGZyb21fY2dpX3BhcmFtcyhsb2NrLCBwYXJhbXMpOgogICAgICAgICAgICByZXR1cm4gX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlcigKICAgICAgICAgICAgICAgIGxvY2s9bG9jaywKICAgICAgICAgICAgICAgIGNvdW50Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgiY291bnQiLCBbTm9uZV0pWzBdLCBpbnQpLAogICAgICAgICAgICAgICAgYWZ0ZXJfPV9hbmRfdGhlbihwYXJhbXMuZ2V0KCJhZnRlciIsIFtOb25lXSlbMF0sIGludCksCiAgICAgICAgICAgICAgICBhY3Rpb25fPXBhcmFtcy5nZXQoImFjdGlvbiIsIFtOb25lXSlbMF0sCiAgICAgICAgICAgICAgICBhY3Rpb25fYXJnc189cGFyYW1zLmdldCgiYWN0aW9uX2FyZ3MiLCBbXSksCiAgICAgICAgICAgICkKCiAgICAgICAgZGVmIF9fc3RyX18oc2VsZik6CiAgICAgICAgICAgIHJldHVybiBmImNvdW50OntzZWxmLmNvdW50fSBhZnRlcjp7c2VsZi5hZnRlcn0gYWN0aW9uOntzZWxmLmFjdGlvbn0gYWN0aW9uX2FyZ3M6e3NlbGYuYWN0aW9uX2FyZ3N9IgoKICAgICAgICBkZWYgaGFzX2VmZmVjdChzZWxmKToKICAgICAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgICAgICBpZiBzZWxmLmFmdGVyOgogICAgICAgICAgICAgICAgICAgIHNlbGYuYWZ0ZXIgLT0gMQogICAgICAgICAgICAgICAgaWYgc2VsZi5hZnRlciA9PSAwOgogICAgICAgICAgICAgICAgICAgIGlmIHNlbGYuY291bnQ6CiAgICAgICAgICAgICAgICAgICAgICAgIHNlbGYuY291bnQgLT0gMQogICAgICAgICAgICAgICAgICAgICAgICByZXR1cm4gVHJ1ZQogICAgICAgICAgICAgICAgcmV0dXJuIEZhbHNlCgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyLmluamVjdF9lcnJvcihyZXF1ZXN0X2hhbmRsZXIpCgogICAgZGVmIF9faW5pdF9fKHNlbGYpOgogICAgICAgIHNlbGYubG9jayA9IHRocmVhZGluZy5Mb2NrKCkKICAgICAgICBzZWxmLmF0X3BhcnRfdXBsb2FkID0gTm9uZQogICAgICAgIHNlbGYuYXRfb2JqZWN0X3VwbG9hZCA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfdXBsb2FkcyA9IGRpY3QoKQogICAgICAgIHNlbGYuc2xvd19wdXQgPSBOb25lCiAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgc2VsZi5hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQgPSBOb25lCgogICAgZGVmIHJlZ2lzdGVyX2Zha2VfdXBsb2FkKHNlbGYsIHVwbG9hZF9pZCwga2V5KToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHNbdXBsb2FkX2lkXSA9IGtleQoKICAgIGRlZiBpc19mYWtlX3VwbG9hZChzZWxmLCB1cGxvYWRfaWQsIGtleSk6CiAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgIGlmIHVwbG9hZF9pZCBpbiBzZWxmLmZha2VfdXBsb2FkczoKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLmZha2VfdXBsb2Fkc1t1cGxvYWRfaWRdID09IGtleQogICAgICAgIHJldHVybiBGYWxzZQoKICAgIGRlZiByZXNldChzZWxmKToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5hdF9wYXJ0X3VwbG9hZCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5hdF9vYmplY3RfdXBsb2FkID0gTm9uZQogICAgICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHMgPSBkaWN0KCkKICAgICAgICAgICAgc2VsZi5zbG93X3B1dCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgICAgIHNlbGYuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkID0gTm9uZQoKCl9ydW50aW1lID0gX1NlcnZlclJ1bnRpbWUoKQoKCmRlZiBnZXRfcmFuZG9tX3N0cmluZyhsZW5ndGgpOgogICAgIyBjaG9vc2UgZnJvbSBhbGwgbG93ZXJjYXNlIGxldHRlcgogICAgbGV0dGVycyA9IHN0cmluZy5hc2NpaV9sb3dlcmNhc2UKICAgIHJlc3VsdF9zdHIgPSAiIi5qb2luKHJhbmRvbS5jaG9pY2UobGV0dGVycykgZm9yIGkgaW4gcmFuZ2UobGVuZ3RoKSkKICAgIHJldHVybiByZXN1bHRfc3RyCgoKY2xhc3MgUmVxdWVzdEhhbmRsZXIoaHR0cC5zZXJ2ZXIuQmFzZUhUVFBSZXF1ZXN0SGFuZGxlcik6CiAgICBkZWYgX29rKHNlbGYpOgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYiJPSyIpCgogICAgZGVmIF9waW5nKHNlbGYpOgogICAgICAgIHNlbGYuX29rKCkKCiAgICBkZWYgcmVhZF9hbGxfaW5wdXQoc2VsZik6CiAgICAgICAgY29udGVudF9sZW5ndGggPSBpbnQoc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiLCAwKSkKICAgICAgICB0b19yZWFkID0gY29udGVudF9sZW5ndGgKICAgICAgICB3aGlsZSB0b19yZWFkID4gMDoKICAgICAgICAgICAgIyByZWFkIGNvbnRlbnQgaW4gb3JkZXIgdG8gYXZvaWQgZXJyb3Igb24gY2xpZW50CiAgICAgICAgICAgICMgUG9jbzo6RXhjZXB0aW9uLiBDb2RlOiAxMDAwLCBlLmNvZGUoKSA9IDMyLCBJL08gZXJyb3I6IEJyb2tlbiBwaXBlCiAgICAgICAgICAgICMgZG8gaXQgcGllY2UgYnkgcGllY2UgaW4gb3JkZXIgdG8gYXZvaWQgYmlnIGFsbG9jYXRpb24KICAgICAgICAgICAgc2l6ZSA9IG1pbih0b19yZWFkLCAxMDI0KQogICAgICAgICAgICBzdHIoc2VsZi5yZmlsZS5yZWFkKHNpemUpKQogICAgICAgICAgICB0b19yZWFkIC09IHNpemUKCiAgICBkZWYgcmVkaXJlY3Qoc2VsZiwgaG9zdD1Ob25lLCBwb3J0PU5vbmUpOgogICAgICAgIGlmIGhvc3QgaXMgTm9uZSBhbmQgcG9ydCBpcyBOb25lOgogICAgICAgICAgICBob3N0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1faG9zdAogICAgICAgICAgICBwb3J0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1fcG9ydAoKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKCiAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDMwNykKICAgICAgICB1cmwgPSBmImh0dHA6Ly97aG9zdH06e3BvcnR9e3NlbGYucGF0aH0iCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgicmVkaXJlY3QgdG8gJXMiLCB1cmwpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiTG9jYXRpb24iLCB1cmwpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIlJlZGlyZWN0ZWQiKQoKICAgIGRlZiB3cml0ZV9lcnJvcihzZWxmLCBkYXRhLCBjb250ZW50X2xlbmd0aD1Ob25lKToKICAgICAgICBpZiBjb250ZW50X2xlbmd0aCBpcyBOb25lOgogICAgICAgICAgICBjb250ZW50X2xlbmd0aCA9IGxlbihkYXRhKQogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoIndyaXRlX2Vycm9yICVzIiwgZGF0YSkKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNTAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBzdHIoY29udGVudF9sZW5ndGgpKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIGlmIGRhdGE6CiAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9mYWtlX3B1dF9vayhzZWxmKToKICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJmYWtlIHB1dCIpCgogICAgICAgIHNlbGYucmVhZF9hbGxfaW5wdXQoKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiRVRhZyIsICJiNTQzNTdmYWYwNjMyY2NlNDZlOTQyZmE2ODM1NmIzOCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCAwKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgIGRlZiBfZmFrZV91cGxvYWRzKHNlbGYsIHBhdGgsIHVwbG9hZF9pZCk6CiAgICAgICAgc2VsZi5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgIHBhcnRzID0gW3ggZm9yIHggaW4gcGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYnVja2V0ID0gcGFydHNbMF0KICAgICAgICBrZXkgPSAiLyIuam9pbihwYXJ0c1sxOl0pCiAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICI8SW5pdGlhdGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgICAgIGYiPEJ1Y2tldD57YnVja2V0fTwvQnVja2V0PiIKICAgICAgICAgICAgZiI8S2V5PntrZXl9PC9LZXk+IgogICAgICAgICAgICBmIjxVcGxvYWRJZD57dXBsb2FkX2lkfTwvVXBsb2FkSWQ+IgogICAgICAgICAgICAiPC9Jbml0aWF0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD4iCiAgICAgICAgKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4oZGF0YSkpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9mYWtlX3Bvc3Rfb2soc2VsZiwgcGF0aCk6CiAgICAgICAgc2VsZi5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgIHBhcnRzID0gW3ggZm9yIHggaW4gcGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYnVja2V0ID0gcGFydHNbMF0KICAgICAgICBrZXkgPSAiLyIuam9pbihwYXJ0c1sxOl0pCiAgICAgICAgbG9jYXRpb24gPSAiaHR0cDovL0V4YW1wbGUtQnVja2V0LnMzLlJlZ2lvbi5hbWF6b25hd3MuY29tLyIgKyBwYXRoCiAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICI8Q29tcGxldGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgICAgIGYiPExvY2F0aW9uPntsb2NhdGlvbn08L0xvY2F0aW9uPlxuIgogICAgICAgICAgICBmIjxCdWNrZXQ+e2J1Y2tldH08L0J1Y2tldD5cbiIKICAgICAgICAgICAgZiI8S2V5PntrZXl9PC9LZXk+XG4iCiAgICAgICAgICAgIGYnPEVUYWc+IjM4NThmNjIyMzBhYzNjOTE1ZjMwMGM2NjQzMTJjMTFmLTkiPC9FVGFnPlxuJwogICAgICAgICAgICBmIjwvQ29tcGxldGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4oZGF0YSkpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9tb2NrX3NldHRpbmdzKHNlbGYpOgogICAgICAgIHBhcnRzID0gdXJsbGliLnBhcnNlLnVybHNwbGl0KHNlbGYucGF0aCkKICAgICAgICBwYXRoID0gW3ggZm9yIHggaW4gcGFydHMucGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYXNzZXJ0IHBhdGhbMF0gPT0gIm1vY2tfc2V0dGluZ3MiLCBwYXRoCiAgICAgICAgaWYgbGVuKHBhdGgpIDwgMjoKICAgICAgICAgICAgcmV0dXJuIHNlbGYud3JpdGVfZXJyb3IoIl9tb2NrX3NldHRpbmdzOiB3cm9uZyBjb21tYW5kIikKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAiYXRfcGFydF91cGxvYWQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCA9IF9TZXJ2ZXJSdW50aW1lLkNvdW50QWZ0ZXIuZnJvbV9jZ2lfcGFyYW1zKAogICAgICAgICAgICAgICAgX3J1bnRpbWUubG9jaywgcGFyYW1zCiAgICAgICAgICAgICkKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IGF0X3BhcnRfdXBsb2FkICVzIiwgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImF0X29iamVjdF91cGxvYWQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkID0gX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlci5mcm9tX2NnaV9wYXJhbXMoCiAgICAgICAgICAgICAgICBfcnVudGltZS5sb2NrLCBwYXJhbXMKICAgICAgICAgICAgKQogICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJzZXQgYXRfb2JqZWN0X3VwbG9hZCAlcyIsIF9ydW50aW1lLmF0X29iamVjdF91cGxvYWQpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImZha2VfcHV0cyI6CiAgICAgICAgICAgIHBhcmFtcyA9IHVybGxpYi5wYXJzZS5wYXJzZV9xcyhwYXJ0cy5xdWVyeSwga2VlcF9ibGFua192YWx1ZXM9RmFsc2UpCiAgICAgICAgICAgIF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IGludCgKICAgICAgICAgICAgICAgIHBhcmFtcy5nZXQoIndoZW5fbGVuZ3RoX2JpZ2dlciIsIFsxMDI0ICogMTAyNF0pWzBdCiAgICAgICAgICAgICkKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IGZha2VfcHV0cyAlcyIsIF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlcikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAic2xvd19wdXQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5zbG93X3B1dCA9IF9TZXJ2ZXJSdW50aW1lLlNsb3dQdXQoCiAgICAgICAgICAgICAgICBsb2NrPV9ydW50aW1lLmxvY2ssCiAgICAgICAgICAgICAgICBtaW5pbWFsX2xlbmd0aF89X2FuZF90aGVuKHBhcmFtcy5nZXQoIm1pbmltYWxfbGVuZ3RoIiwgW05vbmVdKVswXSwgaW50KSwKICAgICAgICAgICAgICAgIHByb2JhYmlsaXR5Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgicHJvYmFiaWxpdHkiLCBbTm9uZV0pWzBdLCBmbG9hdCksCiAgICAgICAgICAgICAgICB0aW1lb3V0Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgidGltZW91dCIsIFtOb25lXSlbMF0sIGZsb2F0KSwKICAgICAgICAgICAgICAgIGNvdW50Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgiY291bnQiLCBbTm9uZV0pWzBdLCBpbnQpLAogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInNldCBzbG93IHB1dCAlcyIsIF9ydW50aW1lLnNsb3dfcHV0KQogICAgICAgICAgICByZXR1cm4gc2VsZi5fb2soKQoKICAgICAgICBpZiBwYXRoWzFdID09ICJzZXR1cF9mYWtlX211bHRwYXJ0dXBsb2FkcyI6CiAgICAgICAgICAgIF9ydW50aW1lLmZha2VfbXVsdGlwYXJ0X3VwbG9hZCA9IFRydWUKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IHNldHVwX2Zha2VfbXVsdHBhcnR1cGxvYWRzIikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAiYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIjoKICAgICAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICAgICAgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkID0gKAogICAgICAgICAgICAgICAgX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlci5mcm9tX2NnaV9wYXJhbXMoX3J1bnRpbWUubG9jaywgcGFyYW1zKQogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICAgICAic2V0IGF0X2NyZWF0ZV9tdWx0aV9wYXJ0X3VwbG9hZCAlcyIsCiAgICAgICAgICAgICAgICBfcnVudGltZS5hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQsCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAicmVzZXQiOgogICAgICAgICAgICBfcnVudGltZS5yZXNldCgpCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInJlc2V0IikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgcmV0dXJuIHNlbGYud3JpdGVfZXJyb3IoIl9tb2NrX3NldHRpbmdzOiB3cm9uZyBjb21tYW5kIikKCiAgICBkZWYgZG9fR0VUKHNlbGYpOgogICAgICAgIGlmIHNlbGYucGF0aCA9PSAiLyI6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9waW5nKCkKCiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9tb2NrX3NldHRpbmdzIik6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9tb2NrX3NldHRpbmdzKCkKCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgiZ2V0IHJlZGlyZWN0IikKICAgICAgICByZXR1cm4gc2VsZi5yZWRpcmVjdCgpCgogICAgZGVmIGRvX1BVVChzZWxmKToKICAgICAgICBjb250ZW50X2xlbmd0aCA9IGludChzZWxmLmhlYWRlcnMuZ2V0KCJDb250ZW50LUxlbmd0aCIsIDApKQoKICAgICAgICBpZiBfcnVudGltZS5zbG93X3B1dCBpcyBub3QgTm9uZToKICAgICAgICAgICAgdGltZW91dCA9IF9ydW50aW1lLnNsb3dfcHV0LmdldF90aW1lb3V0KGNvbnRlbnRfbGVuZ3RoKQogICAgICAgICAgICBpZiB0aW1lb3V0IGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2xvdyBwdXQgJXMiLCB0aW1lb3V0KQogICAgICAgICAgICAgICAgdGltZS5zbGVlcCh0aW1lb3V0KQoKICAgICAgICBwYXJ0cyA9IHVybGxpYi5wYXJzZS51cmxzcGxpdChzZWxmLnBhdGgpCiAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KCiAgICAgICAgaWYgdXBsb2FkX2lkIGlzIG5vdCBOb25lOgogICAgICAgICAgICBpZiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCBpcyBub3QgTm9uZToKICAgICAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICAgICAgICAgInB1dCBhdF9wYXJ0X3VwbG9hZCAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICB1cGxvYWRfaWQsCiAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICApCgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQuaGFzX2VmZmVjdCgpOgogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZC5pbmplY3RfZXJyb3Ioc2VsZikKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuaXNfZmFrZV91cGxvYWQodXBsb2FkX2lkLCBwYXJ0cy5wYXRoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wdXRfb2soKQogICAgICAgIGVsc2U6CiAgICAgICAgICAgIGlmIF9ydW50aW1lLmF0X29iamVjdF91cGxvYWQgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGVycm9yX2F0X29iamVjdF91cGxvYWQgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuYXRfb2JqZWN0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQogICAgICAgICAgICBpZiBfcnVudGltZS5mYWtlX3B1dF93aGVuX2xlbmd0aF9iaWdnZXIgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBjb250ZW50X2xlbmd0aCA+IF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlcjoKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuZmFrZV9wdXRfd2hlbl9sZW5ndGhfYmlnZ2VyLAogICAgICAgICAgICAgICAgICAgICAgICBjb250ZW50X2xlbmd0aCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3B1dF9vaygpCgogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICJwdXQgcmVkaXJlY3QgJXMiLAogICAgICAgICAgICBwYXJ0cywKICAgICAgICApCiAgICAgICAgcmV0dXJuIHNlbGYucmVkaXJlY3QoKQoKICAgIGRlZiBkb19QT1NUKHNlbGYpOgogICAgICAgIHBhcnRzID0gdXJsbGliLnBhcnNlLnVybHNwbGl0KHNlbGYucGF0aCkKICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCiAgICAgICAgdXBsb2FkcyA9IHBhcmFtcy5nZXQoInVwbG9hZHMiLCBbTm9uZV0pWzBdCiAgICAgICAgaWYgdXBsb2FkcyBpcyBub3QgTm9uZToKICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQoKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgdXBsb2FkX2lkID0gZ2V0X3JhbmRvbV9zdHJpbmcoNSkKICAgICAgICAgICAgICAgIF9ydW50aW1lLnJlZ2lzdGVyX2Zha2VfdXBsb2FkKHVwbG9hZF9pZCwgcGFydHMucGF0aCkKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3VwbG9hZHMocGFydHMucGF0aCwgdXBsb2FkX2lkKQoKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KICAgICAgICBpZiBfcnVudGltZS5pc19mYWtlX3VwbG9hZCh1cGxvYWRfaWQsIHBhcnRzLnBhdGgpOgogICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wb3N0X29rKHBhcnRzLnBhdGgpCgogICAgICAgIHJldHVybiBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fSEVBRChzZWxmKToKICAgICAgICBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fREVMRVRFKHNlbGYpOgogICAgICAgIHNlbGYucmVkaXJlY3QoKQoKCmNsYXNzIF9UaHJlYWRlZEhUVFBTZXJ2ZXIoc29ja2V0c2VydmVyLlRocmVhZGluZ01peEluLCBodHRwLnNlcnZlci5IVFRQU2VydmVyKToKICAgICIiIkhhbmRsZSByZXF1ZXN0cyBpbiBhIHNlcGFyYXRlIHRocmVhZC4iIiIKCiAgICBkZWYgc2V0X3Vwc3RyZWFtKHNlbGYsIHVwc3RyZWFtX2hvc3QsIHVwc3RyZWFtX3BvcnQpOgogICAgICAgIHNlbGYudXBzdHJlYW1faG9zdCA9IHVwc3RyZWFtX2hvc3QKICAgICAgICBzZWxmLnVwc3RyZWFtX3BvcnQgPSB1cHN0cmVhbV9wb3J0CgoKaWYgX19uYW1lX18gPT0gIl9fbWFpbl9fIjoKICAgIGh0dHBkID0gX1RocmVhZGVkSFRUUFNlcnZlcigoIjAuMC4wLjAiLCBpbnQoc3lzLmFyZ3ZbMV0pKSwgUmVxdWVzdEhhbmRsZXIpCiAgICBpZiBsZW4oc3lzLmFyZ3YpID09IDQ6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKHN5cy5hcmd2WzJdLCBzeXMuYXJndlszXSkKICAgIGVsc2U6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKCJtaW5pbzEiLCA5MDAxKQogICAgaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > broken_s3.py'] (cluster.py:113, run_and_check) 2026-10-28 23:52:57 [ 499 ] DEBUG : run container_id:roottestcheckings3blobsparanoid_resolver_1 detach:True nothrow:False cmd: ['python3', 'broken_s3.py', '8083'] (cluster.py:2046, exec_in_container) 2026-10-28 23:52:57 [ 499 ] DEBUG : run container_id:roottestcheckings3blobsparanoid_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/'] (cluster.py:2046, exec_in_container) 2026-10-28 23:52:57 [ 499 ] DEBUG : Command:['docker', 'exec', 'roottestcheckings3blobsparanoid_resolver_1', 'curl', '-s', 'http://localhost:8083/'] (cluster.py:113, run_and_check) 2026-10-28 23:52:57 [ 499 ] DEBUG : Exitcode:7 (cluster.py:125, run_and_check) 2026-10-28 23:52:58 [ 499 ] DEBUG : run container_id:roottestcheckings3blobsparanoid_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/'] (cluster.py:2046, exec_in_container) 2026-10-28 23:52:58 [ 499 ] DEBUG : Command:['docker', 'exec', 'roottestcheckings3blobsparanoid_resolver_1', 'curl', '-s', 'http://localhost:8083/'] (cluster.py:113, run_and_check) 2026-10-28 23:52:59 [ 499 ] DEBUG : Stdout:OK (cluster.py:121, run_and_check) 2026-10-28 23:52:59 [ 499 ] DEBUG : broken_s3.py answered OK on attempt 2 (mock_servers.py:53, start_mock_servers) 2026-10-28 23:52:59 [ 499 ] INFO : Mock server broken_s3.py started (mock_servers.py:68, start_mock_servers) 2026-10-28 23:52:59 [ 499 ] DEBUG : run container_id:roottestcheckings3blobsparanoid_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/mock_settings/reset'] (cluster.py:2046, exec_in_container) 2026-10-28 23:52:59 [ 499 ] DEBUG : Command:['docker', 'exec', 'roottestcheckings3blobsparanoid_resolver_1', 'curl', '-s', 'http://localhost:8083/mock_settings/reset'] (cluster.py:113, run_and_check) 2026-10-28 23:52:59 [ 499 ] DEBUG : Stdout:OK (cluster.py:121, run_and_check) ------------------------------ Captured log call ------------------------------- 2026-10-28 23:52:59 [ 499 ] DEBUG : run container_id:roottestcheckings3blobsparanoid_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/mock_settings/at_part_upload?nothing=1&count=10000000&after=2&action=connection_refused'] (cluster.py:2046, exec_in_container) 2026-10-28 23:52:59 [ 499 ] DEBUG : Command:['docker', 'exec', 'roottestcheckings3blobsparanoid_resolver_1', 'curl', '-s', 'http://localhost:8083/mock_settings/at_part_upload?nothing=1&count=10000000&after=2&action=connection_refused'] (cluster.py:113, run_and_check) 2026-10-28 23:52:59 [ 499 ] DEBUG : Stdout:OK (cluster.py:121, run_and_check) 2026-10-28 23:52:59 [ 499 ] DEBUG : Executing query INSERT INTO TABLE FUNCTION s3( 'http://resolver:8083/root/data/test_query_is_canceled_with_inf_retries', 'minio', 'minio123', 'CSV', auto, 'none' ) SELECT * FROM system.numbers LIMIT 1000000 SETTINGS s3_max_single_part_upload_size=100, s3_min_upload_part_size=10000, s3_check_objects_after_upload=0 on node_with_inf_s3_retries (cluster.py:3520, get_query_request) 2026-10-28 23:52:59 [ 499 ] DEBUG : Executing query SELECT count() FROM system.processes WHERE query_id='TEST_QUERY_IS_CANCELED_WITH_INF_RETRIES' on node_with_inf_s3_retries (cluster.py:3455, query) 2026-10-28 23:52:59 [ 499 ] DEBUG : Executing query SELECT ProfileEvents['S3WriteRequestsErrors'] > 10 FROM system.processes WHERE query_id='TEST_QUERY_IS_CANCELED_WITH_INF_RETRIES' on node_with_inf_s3_retries (cluster.py:3455, query) 2026-10-28 23:52:59 [ 499 ] DEBUG : Executing query KILL QUERY WHERE query_id = 'TEST_QUERY_IS_CANCELED_WITH_INF_RETRIES' ASYNC on node_with_inf_s3_retries (cluster.py:3455, query) ---------------------------- Captured log teardown ----------------------------- 2026-10-28 23:53:01 [ 499 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env', '--project-name', 'roottestcheckings3blobsparanoid', '--file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node_with_inf_s3_retries/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:113, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_node_with_inf_s3_retries_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_node_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_minio1_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_resolver_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_proxy2_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_proxy1_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_node_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_node_with_inf_s3_retries_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_minio1_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_resolver_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_proxy1_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Stopping roottestcheckings3blobsparanoid_proxy2_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/logs/stderr.log* || true'] (cluster.py:113, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node_with_inf_s3_retries/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node_with_inf_s3_retries/logs/stderr.log* || true'] (cluster.py:113, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/.env', '--project-name', 'roottestcheckings3blobsparanoid', '--file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_checking_s3_blobs_paranoid/_instances_2/node_with_inf_s3_retries/docker-compose.yml', 'down', '--volumes'] (cluster.py:113, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_node_with_inf_s3_retries_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_node_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_minio1_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_resolver_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_proxy2_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_proxy1_1 ... (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_node_with_inf_s3_retries_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_node_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_resolver_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_proxy1_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_proxy2_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing roottestcheckings3blobsparanoid_minio1_1 ... done (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing network roottestcheckings3blobsparanoid_default (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Stderr:Removing volume roottestcheckings3blobsparanoid_data1-1 (cluster.py:123, run_and_check) 2026-10-28 23:53:22 [ 499 ] DEBUG : Cleanup called (cluster.py:801, cleanup) 2026-10-28 23:53:22 [ 499 ] DEBUG : Docker networks for project roottestcheckings3blobsparanoid are NETWORK ID NAME DRIVER SCOPE (cluster.py:780, print_all_docker_pieces) 2026-10-28 23:53:23 [ 499 ] DEBUG : Docker containers for project roottestcheckings3blobsparanoid are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:788, print_all_docker_pieces) 2026-10-28 23:53:23 [ 499 ] DEBUG : Docker volumes for project roottestcheckings3blobsparanoid are DRIVER VOLUME NAME (cluster.py:796, print_all_docker_pieces) 2026-10-28 23:53:23 [ 499 ] DEBUG : Command:docker container list --all --filter name='^/roottestcheckings3blobsparanoid_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-10-28 23:53:23 [ 499 ] DEBUG : Unstopped containers: {} (cluster.py:815, cleanup) 2026-10-28 23:53:23 [ 499 ] DEBUG : No running containers for project: roottestcheckings3blobsparanoid (cluster.py:829, cleanup) 2026-10-28 23:53:23 [ 499 ] DEBUG : Trying to prune unused networks... (cluster.py:835, cleanup) 2026-10-28 23:53:23 [ 499 ] DEBUG : Trying to prune unused images... (cluster.py:851, cleanup) 2026-10-28 23:53:23 [ 499 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2026-10-28 23:53:23 [ 499 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check) 2026-10-28 23:53:23 [ 499 ] DEBUG : Images pruned (cluster.py:854, cleanup) 2026-10-28 23:53:23 [ 499 ] DEBUG : Trying to prune unused volumes... (cluster.py:860, cleanup) 2026-10-28 23:53:23 [ 499 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-10-28 23:53:23 [ 499 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) ============================== slowest durations =============================== 22.21s teardown test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries 8.73s setup test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries 1.66s call test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries =========================== short test summary info ============================ FAILED test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries ============================== 1 failed in 32.76s ============================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 437, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_tq4w3n --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=2993bc2bf171 -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS=" -rfEps --run-id=2 --color=no --durations=0 test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 ' returned non-zero exit status 1.